Jul 24 17:42:54.833093 ip-10-0-141-191 systemd[1]: Starting Kubernetes Kubelet... Jul 24 17:42:55.281902 ip-10-0-141-191 kubenswrapper[2559]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:42:55.281902 ip-10-0-141-191 kubenswrapper[2559]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 24 17:42:55.281902 ip-10-0-141-191 kubenswrapper[2559]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:42:55.281902 ip-10-0-141-191 kubenswrapper[2559]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 24 17:42:55.281902 ip-10-0-141-191 kubenswrapper[2559]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 24 17:42:55.284083 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.283973 2559 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 24 17:42:55.291187 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291166 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:42:55.291187 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291185 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:42:55.291187 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291189 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291192 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291195 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291198 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291201 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291206 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291209 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291212 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291215 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291218 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291222 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291224 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291227 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291230 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291232 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291235 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291239 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291248 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291251 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:42:55.291282 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291253 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291256 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291259 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291261 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291264 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291267 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291269 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291272 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291274 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291280 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291284 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291287 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291291 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291294 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291297 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291299 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291302 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291306 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291309 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:42:55.291732 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291312 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291314 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291317 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291319 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291322 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291325 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291327 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291329 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291332 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291334 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291337 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291340 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291342 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291344 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291347 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291350 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291353 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291356 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291358 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291361 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:42:55.292208 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291363 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291367 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291370 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291373 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291376 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291379 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291383 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291386 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291389 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291391 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291394 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291397 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291399 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291402 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291404 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291407 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291409 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291412 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291414 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:42:55.292701 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291417 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291419 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291422 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291424 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291427 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291429 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291432 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291434 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291437 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291440 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291443 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291445 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291448 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291450 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291453 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291456 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291458 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291461 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291463 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.291467 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:42:55.293227 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292241 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292249 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292252 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292256 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292259 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292263 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292266 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292269 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292271 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292274 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292277 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292280 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292282 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292285 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292287 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292290 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292292 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292295 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292297 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292300 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:42:55.293704 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292303 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292307 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292310 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292313 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292316 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292319 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292321 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292324 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292327 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292329 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292332 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292334 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292337 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292339 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292343 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292345 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292347 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292350 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292353 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292356 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:42:55.294222 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292358 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292361 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292365 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292367 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292370 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292375 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292379 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292382 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292385 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292387 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292390 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292393 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292396 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292398 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292401 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292404 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292406 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292409 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292411 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292414 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:42:55.294718 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292416 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292428 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292431 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292434 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292437 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292439 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292442 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292444 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292447 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292449 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292452 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292456 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292459 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292461 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292464 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292467 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292470 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292472 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292475 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:42:55.295230 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292477 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292479 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292482 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292486 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292488 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292491 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292496 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292499 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292502 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292505 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292508 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292511 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292514 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292516 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292519 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292522 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292524 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292527 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292529 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292531 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:42:55.295699 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292982 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292988 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292992 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.292994 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293002 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293007 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293010 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293013 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293016 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293019 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293021 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293025 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293029 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293032 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293035 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293038 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293041 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293044 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:42:55.296197 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293047 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293050 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293052 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293055 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293057 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293060 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293062 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293065 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293067 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293070 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293072 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293075 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293077 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293080 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293083 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293086 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293088 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293091 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293093 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293101 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:42:55.296761 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293104 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293106 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293109 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293112 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293114 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293116 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293119 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293122 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293124 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293127 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293130 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293132 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293135 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293137 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293140 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293142 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293145 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293147 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293150 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293152 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:42:55.297266 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293155 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293157 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293160 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293162 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293165 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293167 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293170 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293173 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293175 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293179 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293181 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293183 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293192 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293195 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293197 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293200 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293202 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293205 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293207 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:42:55.297758 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293210 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293213 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293217 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293220 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293223 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293225 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293228 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293231 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293234 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293237 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293239 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293242 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293244 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293247 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293249 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293252 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293254 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293256 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293259 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293262 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:42:55.298261 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293264 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.293266 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293342 2559 flags.go:64] FLAG: --address="0.0.0.0" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293349 2559 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293355 2559 flags.go:64] FLAG: --anonymous-auth="true" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293360 2559 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293375 2559 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293379 2559 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293383 2559 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293387 2559 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293391 2559 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293394 2559 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293397 2559 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293400 2559 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293406 2559 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293409 2559 flags.go:64] FLAG: --cgroup-root="" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293412 2559 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293415 2559 flags.go:64] FLAG: --client-ca-file="" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293418 2559 flags.go:64] FLAG: --cloud-provider="external" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293421 2559 flags.go:64] FLAG: --cluster-dns="[]" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293426 2559 flags.go:64] FLAG: --cluster-domain="" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293429 2559 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293432 2559 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293436 2559 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 24 17:42:55.298742 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293439 2559 flags.go:64] FLAG: --container-log-max-files="5" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293443 2559 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293446 2559 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293451 2559 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293454 2559 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293457 2559 flags.go:64] FLAG: --contention-profiling="false" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293460 2559 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293463 2559 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293466 2559 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293469 2559 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293473 2559 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293476 2559 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293479 2559 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293482 2559 flags.go:64] FLAG: --enable-load-reader="false" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293485 2559 flags.go:64] FLAG: --enable-server="true" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293488 2559 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293493 2559 flags.go:64] FLAG: --event-burst="100" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293496 2559 flags.go:64] FLAG: --event-qps="50" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293499 2559 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293501 2559 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293504 2559 flags.go:64] FLAG: --eviction-hard="" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293508 2559 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293512 2559 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293515 2559 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293518 2559 flags.go:64] FLAG: --eviction-soft="" Jul 24 17:42:55.299327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293521 2559 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293523 2559 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293526 2559 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293529 2559 flags.go:64] FLAG: --experimental-mounter-path="" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293532 2559 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293535 2559 flags.go:64] FLAG: --fail-swap-on="true" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293538 2559 flags.go:64] FLAG: --feature-gates="" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293542 2559 flags.go:64] FLAG: --file-check-frequency="20s" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293545 2559 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293549 2559 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293556 2559 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293559 2559 flags.go:64] FLAG: --healthz-port="10248" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293562 2559 flags.go:64] FLAG: --help="false" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293566 2559 flags.go:64] FLAG: --hostname-override="ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293569 2559 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293572 2559 flags.go:64] FLAG: --http-check-frequency="20s" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293574 2559 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293578 2559 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293581 2559 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293584 2559 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293587 2559 flags.go:64] FLAG: --image-service-endpoint="" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293589 2559 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293592 2559 flags.go:64] FLAG: --kube-api-burst="100" Jul 24 17:42:55.299951 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293596 2559 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293599 2559 flags.go:64] FLAG: --kube-api-qps="50" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293602 2559 flags.go:64] FLAG: --kube-reserved="" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293605 2559 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293607 2559 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293610 2559 flags.go:64] FLAG: --kubelet-cgroups="" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293614 2559 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293617 2559 flags.go:64] FLAG: --lock-file="" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293620 2559 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293623 2559 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293626 2559 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293632 2559 flags.go:64] FLAG: --log-json-split-stream="false" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293634 2559 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293637 2559 flags.go:64] FLAG: --log-text-split-stream="false" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293640 2559 flags.go:64] FLAG: --logging-format="text" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293642 2559 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293646 2559 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293648 2559 flags.go:64] FLAG: --manifest-url="" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293651 2559 flags.go:64] FLAG: --manifest-url-header="" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293658 2559 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293661 2559 flags.go:64] FLAG: --max-open-files="1000000" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293665 2559 flags.go:64] FLAG: --max-pods="110" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293668 2559 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293671 2559 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293674 2559 flags.go:64] FLAG: --memory-manager-policy="None" Jul 24 17:42:55.300536 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293677 2559 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293680 2559 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293683 2559 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293685 2559 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293693 2559 flags.go:64] FLAG: --node-status-max-images="50" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293696 2559 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293699 2559 flags.go:64] FLAG: --oom-score-adj="-999" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293702 2559 flags.go:64] FLAG: --pod-cidr="" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293706 2559 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293711 2559 flags.go:64] FLAG: --pod-manifest-path="" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293714 2559 flags.go:64] FLAG: --pod-max-pids="-1" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293717 2559 flags.go:64] FLAG: --pods-per-core="0" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293720 2559 flags.go:64] FLAG: --port="10250" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293723 2559 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293726 2559 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-01e2fe77b3c26b7b9" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293729 2559 flags.go:64] FLAG: --qos-reserved="" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293732 2559 flags.go:64] FLAG: --read-only-port="10255" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293735 2559 flags.go:64] FLAG: --register-node="true" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293738 2559 flags.go:64] FLAG: --register-with-taints="" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293742 2559 flags.go:64] FLAG: --registry-burst="10" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293744 2559 flags.go:64] FLAG: --registry-qps="5" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293747 2559 flags.go:64] FLAG: --reserved-cpus="" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293750 2559 flags.go:64] FLAG: --reserved-memory="" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293753 2559 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293757 2559 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 24 17:42:55.301358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293760 2559 flags.go:64] FLAG: --rotate-certificates="false" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293764 2559 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293767 2559 flags.go:64] FLAG: --runonce="false" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293769 2559 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293772 2559 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293775 2559 flags.go:64] FLAG: --seccomp-default="false" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293778 2559 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293796 2559 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293799 2559 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293802 2559 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293806 2559 flags.go:64] FLAG: --storage-driver-password="root" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293809 2559 flags.go:64] FLAG: --storage-driver-secure="false" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293812 2559 flags.go:64] FLAG: --storage-driver-table="stats" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293814 2559 flags.go:64] FLAG: --storage-driver-user="root" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293817 2559 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293820 2559 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293824 2559 flags.go:64] FLAG: --system-cgroups="" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293827 2559 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293832 2559 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293835 2559 flags.go:64] FLAG: --tls-cert-file="" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293839 2559 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293844 2559 flags.go:64] FLAG: --tls-min-version="" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293847 2559 flags.go:64] FLAG: --tls-private-key-file="" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293851 2559 flags.go:64] FLAG: --topology-manager-policy="none" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293854 2559 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 24 17:42:55.302474 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293857 2559 flags.go:64] FLAG: --topology-manager-scope="container" Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293860 2559 flags.go:64] FLAG: --v="2" Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293864 2559 flags.go:64] FLAG: --version="false" Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293869 2559 flags.go:64] FLAG: --vmodule="" Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293872 2559 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.293876 2559 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294009 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294013 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294019 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294023 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294026 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294028 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294031 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294034 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294036 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294039 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294041 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294045 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294048 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294051 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:42:55.303441 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294054 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294057 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294060 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294063 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294066 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294069 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294071 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294073 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294076 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294078 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294081 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294083 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294086 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294088 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294090 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294093 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294096 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294098 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294102 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294105 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:42:55.303968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294109 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294112 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294114 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294117 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294119 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294122 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294124 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294127 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294129 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294132 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294134 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294137 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294139 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294142 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294144 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294147 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294149 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294153 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294155 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294157 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:42:55.304726 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294160 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294162 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294165 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294167 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294169 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294172 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294174 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294176 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294179 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294182 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294184 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294187 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294190 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294193 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294195 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294198 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294200 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294203 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294205 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294208 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:42:55.305570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294210 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294212 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294215 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294217 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294220 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294222 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294225 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294229 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294231 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294234 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294237 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294240 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294242 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294245 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294247 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294249 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294252 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294254 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294257 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:42:55.306152 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294260 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294262 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294265 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294267 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294270 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.294273 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.295038 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.303189 2559 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.303210 2559 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.304148 2559 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.304161 2559 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304250 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304259 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304264 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304268 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:42:55.306797 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304273 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304277 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304284 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304289 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304294 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304299 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304303 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304307 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304312 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304316 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304321 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304325 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304329 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304333 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304338 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304342 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304346 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304350 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304354 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:42:55.307423 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304358 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304363 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304367 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304371 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304375 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304380 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304386 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304391 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304395 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304400 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304404 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304408 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304413 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304417 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304432 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304438 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304443 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304447 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304452 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:42:55.308139 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304459 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304464 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304469 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304475 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304480 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304485 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304489 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304494 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304498 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304503 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304507 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304512 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304516 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304520 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304524 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304528 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304533 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304538 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304542 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304547 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:42:55.308626 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304551 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304555 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304560 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304564 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304569 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304573 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304577 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304582 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304586 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304590 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304594 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304598 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304603 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304607 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304611 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304615 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304621 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304625 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304629 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304633 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:42:55.309267 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304637 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304642 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304646 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304650 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304654 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304659 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304663 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304667 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304671 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304675 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304680 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304684 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304688 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304693 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304697 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304701 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:42:55.309778 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304705 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.304714 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304950 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304960 2559 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304965 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304971 2559 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304978 2559 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304984 2559 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304988 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304992 2559 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.304997 2559 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305001 2559 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305005 2559 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305012 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305016 2559 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 24 17:42:55.310221 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305020 2559 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305025 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305029 2559 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305034 2559 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305038 2559 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305042 2559 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305046 2559 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305050 2559 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305055 2559 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305059 2559 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305063 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305067 2559 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305072 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305076 2559 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305080 2559 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305084 2559 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305089 2559 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305093 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305097 2559 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305102 2559 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 24 17:42:55.310604 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305106 2559 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305110 2559 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305114 2559 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305118 2559 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305122 2559 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305126 2559 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305130 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305134 2559 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305138 2559 feature_gate.go:328] unrecognized feature gate: Example2 Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305144 2559 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305150 2559 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305155 2559 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305160 2559 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305165 2559 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305170 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305175 2559 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305179 2559 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305183 2559 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305187 2559 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305191 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 24 17:42:55.311171 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305195 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305200 2559 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305204 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305208 2559 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305213 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305217 2559 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305221 2559 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305225 2559 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305229 2559 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305234 2559 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305238 2559 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305243 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305247 2559 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305251 2559 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305255 2559 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305259 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305264 2559 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305268 2559 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305273 2559 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 24 17:42:55.311677 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305277 2559 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305281 2559 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305285 2559 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305289 2559 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305293 2559 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305300 2559 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305306 2559 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305311 2559 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305316 2559 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305321 2559 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305325 2559 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305330 2559 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305334 2559 feature_gate.go:328] unrecognized feature gate: Example Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305338 2559 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305342 2559 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305346 2559 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305351 2559 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305355 2559 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305359 2559 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 24 17:42:55.312307 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305364 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305368 2559 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305372 2559 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305376 2559 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305380 2559 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305384 2559 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305389 2559 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:55.305393 2559 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.305400 2559 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.305574 2559 server.go:962] "Client rotation is on, will bootstrap in background" Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.308839 2559 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.310621 2559 server.go:1019] "Starting client certificate rotation" Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.310732 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 17:42:55.312961 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.310765 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 24 17:42:55.317476 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.317441 2559 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 17:42:55.320637 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.320614 2559 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 24 17:42:55.332862 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.332843 2559 log.go:25] "Validated CRI v1 runtime API" Jul 24 17:42:55.335227 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.335212 2559 log.go:25] "Validated CRI v1 image API" Jul 24 17:42:55.337134 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.337108 2559 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 24 17:42:55.339767 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.339742 2559 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7c39645b-e599-47da-9e8e-1e2b35ba43d7:/dev/nvme0n1p4 c095b23e-8397-47a4-a8b7-b4fe31800686:/dev/nvme0n1p3] Jul 24 17:42:55.339869 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.339766 2559 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 24 17:42:55.344864 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.344838 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 17:42:55.345118 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.344994 2559 manager.go:216] Machine: {Timestamp:2026-07-24 17:42:55.343522611 +0000 UTC m=+0.383316643 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102551 MemoryCapacity:32812167168 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21647f85af9e61177a5e6326e8ef54 SystemUUID:ec21647f-85af-9e61-177a-5e6326e8ef54 BootID:31f7dcdd-d24f-4d5c-a85f-af41fb4b05a7 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406085632 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:fc:08:83:41 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:fc:08:83:41 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:fb:30:dd:41:d0 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812167168 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 24 17:42:55.345118 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.345117 2559 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 24 17:42:55.345298 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.345283 2559 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 24 17:42:55.346445 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.346414 2559 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 24 17:42:55.346632 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.346449 2559 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-191.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 24 17:42:55.347356 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.347343 2559 topology_manager.go:138] "Creating topology manager with none policy" Jul 24 17:42:55.347429 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.347362 2559 container_manager_linux.go:309] "Creating device plugin manager" Jul 24 17:42:55.347429 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.347388 2559 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 17:42:55.348270 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.348257 2559 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 24 17:42:55.350272 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.350259 2559 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 24 17:42:55.355078 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355062 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 24 17:42:55.355216 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355204 2559 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 24 17:42:55.355278 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355249 2559 kubelet.go:486] "Attempting to sync node with API server" Jul 24 17:42:55.355278 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355264 2559 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 24 17:42:55.355361 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355283 2559 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 24 17:42:55.355361 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355297 2559 kubelet.go:398] "Adding apiserver pod source" Jul 24 17:42:55.355361 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355309 2559 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 24 17:42:55.355695 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355681 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 17:42:55.355760 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.355704 2559 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 24 17:42:55.358701 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.358677 2559 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 24 17:42:55.360539 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.360525 2559 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 24 17:42:55.360594 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.360555 2559 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 24 17:42:55.361381 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361370 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 24 17:42:55.361426 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361386 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 24 17:42:55.361426 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361392 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 24 17:42:55.361426 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361398 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 24 17:42:55.361426 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361406 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 24 17:42:55.361426 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361412 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 24 17:42:55.361426 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361420 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 24 17:42:55.361426 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361425 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 24 17:42:55.361597 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361432 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 24 17:42:55.361597 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361437 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 24 17:42:55.361597 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361445 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 24 17:42:55.361597 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.361450 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 24 17:42:55.362106 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.362096 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 24 17:42:55.362106 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.362105 2559 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 24 17:42:55.365429 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.365414 2559 server.go:1268] "Started kubelet" Jul 24 17:42:55.365592 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.365517 2559 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 24 17:42:55.365684 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.365541 2559 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 24 17:42:55.365684 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.365614 2559 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 24 17:42:55.366302 ip-10-0-141-191 systemd[1]: Started Kubernetes Kubelet. Jul 24 17:42:55.366917 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.366693 2559 server.go:310] "Adding debug handlers to kubelet server" Jul 24 17:42:55.367857 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.367840 2559 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 24 17:42:55.370756 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.370731 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 24 17:42:55.370865 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.370772 2559 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-191.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 24 17:42:55.370952 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.370930 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 24 17:42:55.371255 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.371221 2559 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 24 17:42:55.376530 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.375350 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-191.ec2.internal.18c54ab694edcb7d default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-191.ec2.internal,UID:ip-10-0-141-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-191.ec2.internal,},FirstTimestamp:2026-07-24 17:42:55.365385085 +0000 UTC m=+0.405179117,LastTimestamp:2026-07-24 17:42:55.365385085 +0000 UTC m=+0.405179117,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-191.ec2.internal,}" Jul 24 17:42:55.378991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.378968 2559 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 24 17:42:55.378991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.378983 2559 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 24 17:42:55.379144 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.379075 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:55.379817 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.379779 2559 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 24 17:42:55.379929 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.379779 2559 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 24 17:42:55.379929 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.379903 2559 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 24 17:42:55.380064 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380048 2559 factory.go:153] Registering CRI-O factory Jul 24 17:42:55.380064 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380065 2559 factory.go:223] Registration of the crio container factory successfully Jul 24 17:42:55.380347 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380051 2559 reconstruct.go:97] "Volume reconstruction finished" Jul 24 17:42:55.380347 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380080 2559 reconciler.go:29] "Reconciler: start to sync state" Jul 24 17:42:55.380347 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380119 2559 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 24 17:42:55.380347 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380128 2559 factory.go:55] Registering systemd factory Jul 24 17:42:55.380347 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380137 2559 factory.go:223] Registration of the systemd container factory successfully Jul 24 17:42:55.380347 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380162 2559 factory.go:103] Registering Raw factory Jul 24 17:42:55.380843 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.380736 2559 manager.go:1196] Started watching for new ooms in manager Jul 24 17:42:55.381133 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.381118 2559 manager.go:349] Starting recovery of all containers Jul 24 17:42:55.381747 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.381705 2559 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-191.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 24 17:42:55.381844 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.381807 2559 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 24 17:42:55.392859 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.392835 2559 manager.go:354] Recovery completed Jul 24 17:42:55.394047 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.394024 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vdkwv" Jul 24 17:42:55.397337 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.397325 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:42:55.400167 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.400042 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:42:55.400236 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.400183 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:42:55.400236 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.400195 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:42:55.400795 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.400764 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:42:55.400865 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.400763 2559 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 24 17:42:55.400865 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.400811 2559 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 24 17:42:55.400865 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.400827 2559 state_mem.go:36] "Initialized new in-memory state store" Jul 24 17:42:55.401442 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.401423 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:42:55.401511 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.401455 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:42:55.401511 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.401471 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:42:55.401667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.401652 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-vdkwv" Jul 24 17:42:55.403023 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.403009 2559 policy_none.go:49] "None policy: Start" Jul 24 17:42:55.403102 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.403031 2559 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 24 17:42:55.403102 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.403041 2559 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 24 17:42:55.405059 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.405047 2559 policy_none.go:47] "Start" Jul 24 17:42:55.408117 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.408048 2559 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-191.ec2.internal.18c54ab697008d38 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-191.ec2.internal,UID:ip-10-0-141-191.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-191.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-191.ec2.internal,},FirstTimestamp:2026-07-24 17:42:55.40016876 +0000 UTC m=+0.439962795,LastTimestamp:2026-07-24 17:42:55.40016876 +0000 UTC m=+0.439962795,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-191.ec2.internal,}" Jul 24 17:42:55.446415 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.446395 2559 manager.go:337] "Starting Device Plugin manager" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.446443 2559 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.446456 2559 server.go:87] "Starting device plugin registration server" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.446742 2559 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.446753 2559 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.446845 2559 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.446932 2559 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.446940 2559 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.447665 2559 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 24 17:42:55.462345 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.447709 2559 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:55.486108 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.486074 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 24 17:42:55.487526 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.487499 2559 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 24 17:42:55.487635 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.487536 2559 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 24 17:42:55.487635 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.487558 2559 kubelet.go:2448] "Starting kubelet main sync loop" Jul 24 17:42:55.487635 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.487592 2559 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 24 17:42:55.489759 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.489739 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:42:55.547206 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.547126 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:42:55.548164 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.548146 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:42:55.548269 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.548181 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:42:55.548269 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.548194 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:42:55.548269 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.548227 2559 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.559957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.559937 2559 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.560060 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.559966 2559 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-191.ec2.internal\": node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:55.585336 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.585312 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:55.588694 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.588674 2559 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal"] Jul 24 17:42:55.588753 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.588737 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:42:55.590239 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.590219 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:42:55.590313 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.590250 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:42:55.590313 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.590261 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:42:55.592767 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.592752 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:42:55.592894 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.592876 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.592941 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.592930 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:42:55.593442 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.593424 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:42:55.593534 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.593451 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:42:55.593534 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.593460 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:42:55.593624 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.593550 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:42:55.593624 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.593573 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:42:55.593624 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.593588 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:42:55.595816 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.595802 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.595892 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.595826 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:42:55.596471 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.596457 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:42:55.596548 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.596485 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:42:55.596548 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.596497 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:42:55.624540 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.624512 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-191.ec2.internal\" not found" node="ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.629563 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.629543 2559 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-191.ec2.internal\" not found" node="ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.681354 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.681326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/25d7561822a46a6ad06fa6a75d9872a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal\" (UID: \"25d7561822a46a6ad06fa6a75d9872a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.681354 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.681355 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25d7561822a46a6ad06fa6a75d9872a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal\" (UID: \"25d7561822a46a6ad06fa6a75d9872a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.681542 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.681372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2a76f45062ee97c58334419b354384f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-191.ec2.internal\" (UID: \"d2a76f45062ee97c58334419b354384f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.686433 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.686416 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:55.782062 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.782026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2a76f45062ee97c58334419b354384f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-191.ec2.internal\" (UID: \"d2a76f45062ee97c58334419b354384f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.782245 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.782083 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/25d7561822a46a6ad06fa6a75d9872a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal\" (UID: \"25d7561822a46a6ad06fa6a75d9872a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.782245 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.782121 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25d7561822a46a6ad06fa6a75d9872a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal\" (UID: \"25d7561822a46a6ad06fa6a75d9872a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.782245 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.782127 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d2a76f45062ee97c58334419b354384f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-191.ec2.internal\" (UID: \"d2a76f45062ee97c58334419b354384f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.782245 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.782165 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/25d7561822a46a6ad06fa6a75d9872a7-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal\" (UID: \"25d7561822a46a6ad06fa6a75d9872a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.782245 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.782191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/25d7561822a46a6ad06fa6a75d9872a7-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal\" (UID: \"25d7561822a46a6ad06fa6a75d9872a7\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.787149 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.787126 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:55.888048 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.888020 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:55.927243 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.927198 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.937597 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:55.937573 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" Jul 24 17:42:55.988308 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:55.988264 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:56.088900 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:56.088862 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:56.189465 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:56.189372 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:56.290076 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:56.290046 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:56.305420 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.305390 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:42:56.309398 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.309375 2559 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 24 17:42:56.309551 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.309531 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 17:42:56.309551 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.309534 2559 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 24 17:42:56.390259 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:56.390230 2559 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-191.ec2.internal\" not found" Jul 24 17:42:56.407573 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.407542 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 17:37:55 +0000 UTC" deadline="2028-01-12 03:19:40.843116182 +0000 UTC" Jul 24 17:42:56.407573 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.407566 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12873h36m44.435553211s" Jul 24 17:42:56.466940 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.466873 2559 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 24 17:42:56.468169 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.468154 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientMemory" Jul 24 17:42:56.468243 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.468187 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasNoDiskPressure" Jul 24 17:42:56.468243 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.468196 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeHasSufficientPID" Jul 24 17:42:56.480023 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.480004 2559 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:42:56.503072 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.503045 2559 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:42:56.536602 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:56.536550 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25d7561822a46a6ad06fa6a75d9872a7.slice/crio-3bfd58012f886d82b713658c6ed27f06cbf8b39c8ab5bd7439402b6977f0f1c4 WatchSource:0}: Error finding container 3bfd58012f886d82b713658c6ed27f06cbf8b39c8ab5bd7439402b6977f0f1c4: Status 404 returned error can't find the container with id 3bfd58012f886d82b713658c6ed27f06cbf8b39c8ab5bd7439402b6977f0f1c4 Jul 24 17:42:56.537635 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:56.537610 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2a76f45062ee97c58334419b354384f.slice/crio-41e60dd9f25202dd9e434a68ff89ef61f1e16a0dc5a0499aaa7656a6a50910bd WatchSource:0}: Error finding container 41e60dd9f25202dd9e434a68ff89ef61f1e16a0dc5a0499aaa7656a6a50910bd: Status 404 returned error can't find the container with id 41e60dd9f25202dd9e434a68ff89ef61f1e16a0dc5a0499aaa7656a6a50910bd Jul 24 17:42:56.543914 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.543898 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:42:56.580157 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.580112 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" Jul 24 17:42:56.604229 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.604201 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 17:42:56.605208 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.605194 2559 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" Jul 24 17:42:56.619746 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:56.619725 2559 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 24 17:42:57.190058 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.190027 2559 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:42:57.356282 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.356237 2559 apiserver.go:52] "Watching apiserver" Jul 24 17:42:57.364777 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.364735 2559 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 24 17:42:57.365417 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.365383 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4","openshift-cluster-node-tuning-operator/tuned-mjdtk","openshift-dns/node-resolver-gnjpf","openshift-multus/multus-29ff9","openshift-ovn-kubernetes/ovnkube-node-bk2sq","kube-system/global-pull-secret-syncer-zh5f7","kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal","openshift-image-registry/node-ca-fsf7z","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal","openshift-multus/multus-additional-cni-plugins-96j5z","openshift-multus/network-metrics-daemon-5pvds","openshift-network-diagnostics/network-check-target-26mkw","openshift-network-operator/iptables-alerter-tlgtf","kube-system/konnectivity-agent-zkd24"] Jul 24 17:42:57.368306 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.368283 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.371104 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.370761 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.372793 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.371407 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 24 17:42:57.372793 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.371410 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 24 17:42:57.372793 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.371639 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-pxh8b\"" Jul 24 17:42:57.372793 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.371746 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 24 17:42:57.372793 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.372012 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 24 17:42:57.372793 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.372049 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 24 17:42:57.375438 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.375412 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:42:57.375697 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.375677 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 24 17:42:57.375802 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.375717 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-sdkc7\"" Jul 24 17:42:57.375907 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.375891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:57.380930 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.380907 2559 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 24 17:42:57.381542 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.375971 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:42:57.382078 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.382056 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.384650 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.384597 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.386528 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.385914 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 24 17:42:57.386528 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.386058 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5mznr\"" Jul 24 17:42:57.386832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.386812 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.387163 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.387090 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:42:57.387996 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.387464 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 24 17:42:57.387996 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.387731 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b4kjv\"" Jul 24 17:42:57.387996 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.387788 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 24 17:42:57.388842 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.388817 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 24 17:42:57.389080 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.389059 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 24 17:42:57.389343 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.389319 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 24 17:42:57.389694 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.389676 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.390139 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390115 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-node-log\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390253 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-cnibin\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.390253 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390195 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-netns\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.390253 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390219 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-slash\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390427 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390259 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-tuned\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.390427 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-etc-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390427 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-hostroot\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.390427 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390366 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlhbm\" (UniqueName: \"kubernetes.io/projected/0243df16-ec23-48c9-9451-34655a6cee2b-kube-api-access-hlhbm\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390427 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-k8s-cni-cncf-io\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.390427 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390421 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-log-socket\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390451 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390477 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0243df16-ec23-48c9-9451-34655a6cee2b-ovn-node-metrics-cert\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390500 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-cni-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390525 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-conf-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390575 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-ovnkube-script-lib\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390602 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8plzr\" (UniqueName: \"kubernetes.io/projected/8bcdff46-acf9-4858-aee1-d92f3219d7e9-kube-api-access-8plzr\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn65r\" (UniqueName: \"kubernetes.io/projected/0cab09e2-baba-4a01-a58f-53c91600107f-kube-api-access-sn65r\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-env-overrides\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390701 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysctl-conf\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.390773 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390727 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-daemon-config\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390847 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-var-lib-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390883 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-cni-binary-copy\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390912 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390965 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54xgf\" (UniqueName: \"kubernetes.io/projected/96cdd753-0f32-40fa-b472-2dfd5864c909-kube-api-access-54xgf\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.390989 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-kubelet\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391010 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391015 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-systemd\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391040 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-ovnkube-config\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391064 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-os-release\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-socket-dir-parent\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391184 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-system-cni-dir\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391213 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-cni-bin\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391238 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysctl-d\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391262 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-systemd\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391284 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-cnibin\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.391518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391310 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-multus-certs\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391335 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-kubelet\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391379 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-systemd-units\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391411 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391441 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-host\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391480 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-sys\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391506 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-var-lib-kubelet\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391531 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-cni-netd\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391561 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-run\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391587 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96cdd753-0f32-40fa-b472-2dfd5864c909-tmp\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391614 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk7gm\" (UniqueName: \"kubernetes.io/projected/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-kube-api-access-wk7gm\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391676 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-system-cni-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391705 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-etc-kubernetes\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391729 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-ovn\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391759 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-os-release\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391907 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.392564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391948 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-cni-bin\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.391977 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-kubernetes\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392004 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-lib-modules\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392032 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bcdff46-acf9-4858-aee1-d92f3219d7e9-cni-binary-copy\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392056 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-modprobe-d\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392081 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysconfig\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392107 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-cni-multus\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392137 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-run-netns\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392329 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392562 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392704 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.392881 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 24 17:42:57.393525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.393074 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8xnsg\"" Jul 24 17:42:57.394551 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.394531 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.396030 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.395233 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 24 17:42:57.396030 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.395250 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 24 17:42:57.396030 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.395537 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jcfc6\"" Jul 24 17:42:57.396030 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.395834 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 24 17:42:57.397625 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.396675 2559 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 24 17:42:57.397625 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.397324 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:42:57.397625 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.397400 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:42:57.398906 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.398685 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 24 17:42:57.398906 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.398740 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v4s4s\"" Jul 24 17:42:57.399060 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.398975 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 24 17:42:57.400249 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.400201 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.403248 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.402828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:42:57.403905 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.403650 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:42:57.403905 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.403650 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-n5tzf\"" Jul 24 17:42:57.403905 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.403764 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 24 17:42:57.404222 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.404204 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 24 17:42:57.409395 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.409251 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4bdwg\"" Jul 24 17:42:57.409968 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.409613 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 24 17:42:57.409968 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.409842 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 24 17:42:57.425379 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.425334 2559 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtrvc" Jul 24 17:42:57.434914 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.434888 2559 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qtrvc" Jul 24 17:42:57.481723 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.481700 2559 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 24 17:42:57.492406 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-tuned\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.492525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492417 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-etc-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.492525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492453 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73a09260-b421-4b83-8e67-4c52d34fd50c-iptables-alerter-script\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.492525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492482 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73a09260-b421-4b83-8e67-4c52d34fd50c-host-slash\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.492525 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492507 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3076bbf1-a934-4c5d-a679-a3aa1c824daf-tmp-dir\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492534 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-hostroot\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlhbm\" (UniqueName: \"kubernetes.io/projected/0243df16-ec23-48c9-9451-34655a6cee2b-kube-api-access-hlhbm\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492580 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-etc-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-socket-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492651 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-k8s-cni-cncf-io\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492657 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-hostroot\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492688 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-log-socket\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492713 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-k8s-cni-cncf-io\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492717 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492760 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-run-ovn-kubernetes\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492728 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-log-socket\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492760 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0243df16-ec23-48c9-9451-34655a6cee2b-ovn-node-metrics-cert\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.492891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492890 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-cni-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492919 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-conf-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492948 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492974 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-kubelet-config\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-conf-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.492997 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-ovnkube-script-lib\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493041 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493051 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-cni-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8plzr\" (UniqueName: \"kubernetes.io/projected/8bcdff46-acf9-4858-aee1-d92f3219d7e9-kube-api-access-8plzr\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493098 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn65r\" (UniqueName: \"kubernetes.io/projected/0cab09e2-baba-4a01-a58f-53c91600107f-kube-api-access-sn65r\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493121 2559 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493129 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-env-overrides\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493154 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysctl-conf\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-daemon-config\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493208 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-var-lib-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493195 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" event={"ID":"25d7561822a46a6ad06fa6a75d9872a7","Type":"ContainerStarted","Data":"3bfd58012f886d82b713658c6ed27f06cbf8b39c8ab5bd7439402b6977f0f1c4"} Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493232 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-cni-binary-copy\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.493613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493288 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjpq8\" (UniqueName: \"kubernetes.io/projected/b00377e3-0fe2-4fe7-89cb-c892e233959f-kube-api-access-vjpq8\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493326 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493370 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493456 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-tuning-conf-dir\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493472 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-var-lib-openvswitch\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493471 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54xgf\" (UniqueName: \"kubernetes.io/projected/96cdd753-0f32-40fa-b472-2dfd5864c909-kube-api-access-54xgf\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493487 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysctl-conf\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493510 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-kubelet\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493541 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-systemd\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.493558 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493565 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-ovnkube-config\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493619 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-systemd\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493624 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-ovnkube-script-lib\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.493660 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs podName:1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a nodeName:}" failed. No retries permitted until 2026-07-24 17:42:57.993637813 +0000 UTC m=+3.033431835 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs") pod "network-metrics-daemon-5pvds" (UID: "1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493680 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c63c5161-5c4c-44e8-991b-603b5c3b24b9-konnectivity-ca\") pod \"konnectivity-agent-zkd24\" (UID: \"c63c5161-5c4c-44e8-991b-603b5c3b24b9\") " pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.494483 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493738 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-os-release\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493767 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-socket-dir-parent\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493813 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-system-cni-dir\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.493842 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c63c5161-5c4c-44e8-991b-603b5c3b24b9-agent-certs\") pod \"konnectivity-agent-zkd24\" (UID: \"c63c5161-5c4c-44e8-991b-603b5c3b24b9\") " pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494059 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-env-overrides\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494085 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00377e3-0fe2-4fe7-89cb-c892e233959f-host\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3076bbf1-a934-4c5d-a679-a3aa1c824daf-hosts-file\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494143 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dvdl\" (UniqueName: \"kubernetes.io/projected/3076bbf1-a934-4c5d-a679-a3aa1c824daf-kube-api-access-9dvdl\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494175 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-cni-bin\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-cni-binary-copy\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494201 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysctl-d\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494226 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-systemd\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-device-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494114 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-daemon-config\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-kubelet\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494375 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-system-cni-dir\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494590 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9l7pb\" (UniqueName: \"kubernetes.io/projected/3236bb71-d19a-401c-bafc-fbb06ed145fe-kube-api-access-9l7pb\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.495330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494627 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqgdq\" (UniqueName: \"kubernetes.io/projected/73a09260-b421-4b83-8e67-4c52d34fd50c-kube-api-access-xqgdq\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-dbus\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-os-release\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-cni-bin\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-cnibin\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494722 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-systemd\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494706 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysctl-d\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494734 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-multus-certs\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494736 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494762 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-kubelet\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-multus-socket-dir-parent\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494786 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-multus-certs\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494795 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-systemd-units\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494807 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-cnibin\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-kubelet\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494832 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-systemd-units\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494850 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/0243df16-ec23-48c9-9451-34655a6cee2b-ovnkube-config\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494946 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.495957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494984 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-sys-fs\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.494998 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495012 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-host\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495044 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-sys\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495053 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-host\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495069 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-var-lib-kubelet\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495094 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-cni-netd\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495105 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-sys\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495118 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-run\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495141 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96cdd753-0f32-40fa-b472-2dfd5864c909-tmp\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495153 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-cni-netd\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495169 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk7gm\" (UniqueName: \"kubernetes.io/projected/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-kube-api-access-wk7gm\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495168 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-var-lib-kubelet\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495194 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-system-cni-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495201 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-run\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495219 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-etc-kubernetes\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-ovn\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-os-release\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.496735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495305 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495331 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-cni-bin\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495356 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-kubernetes\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495379 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-lib-modules\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bcdff46-acf9-4858-aee1-d92f3219d7e9-cni-binary-copy\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495430 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-registration-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495457 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-modprobe-d\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495493 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysconfig\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495521 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-cni-multus\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495525 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-etc-kubernetes\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495546 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-run-netns\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495573 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-node-log\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495582 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-system-cni-dir\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495599 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-cnibin\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495636 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-run-ovn\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495657 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495685 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:42:57.497696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495719 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-sysconfig\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-var-lib-cni-multus\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495727 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-netns\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495766 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8bcdff46-acf9-4858-aee1-d92f3219d7e9-host-run-netns\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495773 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-cni-bin\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495768 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-slash\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495798 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-slash\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495825 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-host-run-netns\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495829 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-kubernetes\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495881 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b00377e3-0fe2-4fe7-89cb-c892e233959f-serviceca\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495908 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-cnibin\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495942 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/0cab09e2-baba-4a01-a58f-53c91600107f-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495993 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-modprobe-d\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.495883 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/0cab09e2-baba-4a01-a58f-53c91600107f-os-release\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.496009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/0243df16-ec23-48c9-9451-34655a6cee2b-node-log\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.496060 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/96cdd753-0f32-40fa-b472-2dfd5864c909-lib-modules\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.496365 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8bcdff46-acf9-4858-aee1-d92f3219d7e9-cni-binary-copy\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.498639 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.496920 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/0243df16-ec23-48c9-9451-34655a6cee2b-ovn-node-metrics-cert\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.499585 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.498261 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" event={"ID":"d2a76f45062ee97c58334419b354384f","Type":"ContainerStarted","Data":"41e60dd9f25202dd9e434a68ff89ef61f1e16a0dc5a0499aaa7656a6a50910bd"} Jul 24 17:42:57.502571 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.502411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/96cdd753-0f32-40fa-b472-2dfd5864c909-tmp\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.503128 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.502998 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/96cdd753-0f32-40fa-b472-2dfd5864c909-etc-tuned\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.505047 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.504990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlhbm\" (UniqueName: \"kubernetes.io/projected/0243df16-ec23-48c9-9451-34655a6cee2b-kube-api-access-hlhbm\") pod \"ovnkube-node-bk2sq\" (UID: \"0243df16-ec23-48c9-9451-34655a6cee2b\") " pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.505417 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.505376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54xgf\" (UniqueName: \"kubernetes.io/projected/96cdd753-0f32-40fa-b472-2dfd5864c909-kube-api-access-54xgf\") pod \"tuned-mjdtk\" (UID: \"96cdd753-0f32-40fa-b472-2dfd5864c909\") " pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.506017 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.505764 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8plzr\" (UniqueName: \"kubernetes.io/projected/8bcdff46-acf9-4858-aee1-d92f3219d7e9-kube-api-access-8plzr\") pod \"multus-29ff9\" (UID: \"8bcdff46-acf9-4858-aee1-d92f3219d7e9\") " pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.506197 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.506175 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn65r\" (UniqueName: \"kubernetes.io/projected/0cab09e2-baba-4a01-a58f-53c91600107f-kube-api-access-sn65r\") pod \"multus-additional-cni-plugins-96j5z\" (UID: \"0cab09e2-baba-4a01-a58f-53c91600107f\") " pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.507819 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.507792 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk7gm\" (UniqueName: \"kubernetes.io/projected/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-kube-api-access-wk7gm\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596684 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-sys-fs\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596736 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-registration-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596764 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596788 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596813 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-sys-fs\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596837 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b00377e3-0fe2-4fe7-89cb-c892e233959f-serviceca\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596862 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596884 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73a09260-b421-4b83-8e67-4c52d34fd50c-iptables-alerter-script\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596905 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73a09260-b421-4b83-8e67-4c52d34fd50c-host-slash\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3076bbf1-a934-4c5d-a679-a3aa1c824daf-tmp-dir\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-socket-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596978 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-registration-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596989 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-kubelet-config\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597035 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjpq8\" (UniqueName: \"kubernetes.io/projected/b00377e3-0fe2-4fe7-89cb-c892e233959f-kube-api-access-vjpq8\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c63c5161-5c4c-44e8-991b-603b5c3b24b9-konnectivity-ca\") pod \"konnectivity-agent-zkd24\" (UID: \"c63c5161-5c4c-44e8-991b-603b5c3b24b9\") " pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.598683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597117 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c63c5161-5c4c-44e8-991b-603b5c3b24b9-agent-certs\") pod \"konnectivity-agent-zkd24\" (UID: \"c63c5161-5c4c-44e8-991b-603b5c3b24b9\") " pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00377e3-0fe2-4fe7-89cb-c892e233959f-host\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597161 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3076bbf1-a934-4c5d-a679-a3aa1c824daf-hosts-file\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597187 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dvdl\" (UniqueName: \"kubernetes.io/projected/3076bbf1-a934-4c5d-a679-a3aa1c824daf-kube-api-access-9dvdl\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597215 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-device-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597242 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9l7pb\" (UniqueName: \"kubernetes.io/projected/3236bb71-d19a-401c-bafc-fbb06ed145fe-kube-api-access-9l7pb\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597268 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqgdq\" (UniqueName: \"kubernetes.io/projected/73a09260-b421-4b83-8e67-4c52d34fd50c-kube-api-access-xqgdq\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597292 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-dbus\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597453 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-dbus\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597510 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/73a09260-b421-4b83-8e67-4c52d34fd50c-host-slash\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597575 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/73a09260-b421-4b83-8e67-4c52d34fd50c-iptables-alerter-script\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597773 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3076bbf1-a934-4c5d-a679-a3aa1c824daf-tmp-dir\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.596931 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-etc-selinux\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.597944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-socket-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.598013 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-kubelet-config\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.598233 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b00377e3-0fe2-4fe7-89cb-c892e233959f-serviceca\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.598297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-kubelet-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.598755 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c63c5161-5c4c-44e8-991b-603b5c3b24b9-konnectivity-ca\") pod \"konnectivity-agent-zkd24\" (UID: \"c63c5161-5c4c-44e8-991b-603b5c3b24b9\") " pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:42:57.599667 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.598850 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:42:57.600304 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.598892 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b00377e3-0fe2-4fe7-89cb-c892e233959f-host\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.600304 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.598933 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret podName:b88c2b38-14f5-4ef4-8e28-76663ce9e34f nodeName:}" failed. No retries permitted until 2026-07-24 17:42:58.098913016 +0000 UTC m=+3.138707038 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret") pod "global-pull-secret-syncer-zh5f7" (UID: "b88c2b38-14f5-4ef4-8e28-76663ce9e34f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:42:57.600304 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.598961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/3076bbf1-a934-4c5d-a679-a3aa1c824daf-hosts-file\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.600304 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.599094 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3236bb71-d19a-401c-bafc-fbb06ed145fe-device-dir\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.601859 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.601827 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c63c5161-5c4c-44e8-991b-603b5c3b24b9-agent-certs\") pod \"konnectivity-agent-zkd24\" (UID: \"c63c5161-5c4c-44e8-991b-603b5c3b24b9\") " pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:42:57.608575 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.608547 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:42:57.608575 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.608578 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:42:57.608766 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.608593 2559 projected.go:196] Error preparing data for projected volume kube-api-access-dzcln for pod openshift-network-diagnostics/network-check-target-26mkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:42:57.608766 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:57.608661 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln podName:9aeff889-3a93-4ecb-a76d-925dfa85e4fa nodeName:}" failed. No retries permitted until 2026-07-24 17:42:58.108639857 +0000 UTC m=+3.148433893 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dzcln" (UniqueName: "kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln") pod "network-check-target-26mkw" (UID: "9aeff889-3a93-4ecb-a76d-925dfa85e4fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:42:57.610816 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.610788 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjpq8\" (UniqueName: \"kubernetes.io/projected/b00377e3-0fe2-4fe7-89cb-c892e233959f-kube-api-access-vjpq8\") pod \"node-ca-fsf7z\" (UID: \"b00377e3-0fe2-4fe7-89cb-c892e233959f\") " pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.611552 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.611526 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9l7pb\" (UniqueName: \"kubernetes.io/projected/3236bb71-d19a-401c-bafc-fbb06ed145fe-kube-api-access-9l7pb\") pod \"aws-ebs-csi-driver-node-l9zr4\" (UID: \"3236bb71-d19a-401c-bafc-fbb06ed145fe\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.612218 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.612191 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dvdl\" (UniqueName: \"kubernetes.io/projected/3076bbf1-a934-4c5d-a679-a3aa1c824daf-kube-api-access-9dvdl\") pod \"node-resolver-gnjpf\" (UID: \"3076bbf1-a934-4c5d-a679-a3aa1c824daf\") " pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.613105 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.613085 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqgdq\" (UniqueName: \"kubernetes.io/projected/73a09260-b421-4b83-8e67-4c52d34fd50c-kube-api-access-xqgdq\") pod \"iptables-alerter-tlgtf\" (UID: \"73a09260-b421-4b83-8e67-4c52d34fd50c\") " pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:57.694092 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.694009 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-96j5z" Jul 24 17:42:57.701419 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.701394 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" Jul 24 17:42:57.714008 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.713988 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-29ff9" Jul 24 17:42:57.722695 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.722671 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:42:57.730828 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.730806 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:42:57.738968 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.738938 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fsf7z" Jul 24 17:42:57.746957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.746939 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" Jul 24 17:42:57.760766 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.760737 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gnjpf" Jul 24 17:42:57.769669 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:57.769652 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tlgtf" Jul 24 17:42:58.001031 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:58.000949 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:58.001184 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:58.001105 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:42:58.001253 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:58.001188 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs podName:1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a nodeName:}" failed. No retries permitted until 2026-07-24 17:42:59.001167238 +0000 UTC m=+4.040961265 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs") pod "network-metrics-daemon-5pvds" (UID: "1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:42:58.102373 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:58.102333 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:58.102549 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:58.102472 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:42:58.102549 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:58.102546 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret podName:b88c2b38-14f5-4ef4-8e28-76663ce9e34f nodeName:}" failed. No retries permitted until 2026-07-24 17:42:59.102531318 +0000 UTC m=+4.142325340 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret") pod "global-pull-secret-syncer-zh5f7" (UID: "b88c2b38-14f5-4ef4-8e28-76663ce9e34f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:42:58.203563 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:58.203524 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:42:58.203764 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:58.203739 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:42:58.203849 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:58.203776 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:42:58.203849 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:58.203787 2559 projected.go:196] Error preparing data for projected volume kube-api-access-dzcln for pod openshift-network-diagnostics/network-check-target-26mkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:42:58.203940 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:58.203860 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln podName:9aeff889-3a93-4ecb-a76d-925dfa85e4fa nodeName:}" failed. No retries permitted until 2026-07-24 17:42:59.203842341 +0000 UTC m=+4.243636366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzcln" (UniqueName: "kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln") pod "network-check-target-26mkw" (UID: "9aeff889-3a93-4ecb-a76d-925dfa85e4fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:42:58.440212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:58.440144 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:37:57 +0000 UTC" deadline="2028-01-16 19:11:56.813985018 +0000 UTC" Jul 24 17:42:58.440212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:58.440210 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12985h28m58.373780179s" Jul 24 17:42:58.733387 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:58.733000 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96cdd753_0f32_40fa_b472_2dfd5864c909.slice/crio-a1f03c3d2b1d88939993b9282959bfc3922195890efd62af181bd3c3f5b1c502 WatchSource:0}: Error finding container a1f03c3d2b1d88939993b9282959bfc3922195890efd62af181bd3c3f5b1c502: Status 404 returned error can't find the container with id a1f03c3d2b1d88939993b9282959bfc3922195890efd62af181bd3c3f5b1c502 Jul 24 17:42:58.737968 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:58.737924 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3236bb71_d19a_401c_bafc_fbb06ed145fe.slice/crio-c13613ea77d4f73ea9e912ecd086558edb1beaa53d806634f4d96c289dbf9059 WatchSource:0}: Error finding container c13613ea77d4f73ea9e912ecd086558edb1beaa53d806634f4d96c289dbf9059: Status 404 returned error can't find the container with id c13613ea77d4f73ea9e912ecd086558edb1beaa53d806634f4d96c289dbf9059 Jul 24 17:42:58.739473 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:58.739445 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc63c5161_5c4c_44e8_991b_603b5c3b24b9.slice/crio-1da49a7792aab26871db35c86ce7825cafff80cc7c3bdbd0c12fa36ec3e605b4 WatchSource:0}: Error finding container 1da49a7792aab26871db35c86ce7825cafff80cc7c3bdbd0c12fa36ec3e605b4: Status 404 returned error can't find the container with id 1da49a7792aab26871db35c86ce7825cafff80cc7c3bdbd0c12fa36ec3e605b4 Jul 24 17:42:58.740575 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:58.740498 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a09260_b421_4b83_8e67_4c52d34fd50c.slice/crio-d8012009c6d9df2fadd9cb3b56d1b007d1b44cfc280c4d8c446cb41a42092cc3 WatchSource:0}: Error finding container d8012009c6d9df2fadd9cb3b56d1b007d1b44cfc280c4d8c446cb41a42092cc3: Status 404 returned error can't find the container with id d8012009c6d9df2fadd9cb3b56d1b007d1b44cfc280c4d8c446cb41a42092cc3 Jul 24 17:42:58.743890 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:58.743511 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0243df16_ec23_48c9_9451_34655a6cee2b.slice/crio-0b9af210d5426b0b53c7928002b1f9d2df4e3dc6f8257920ba891ffca4c2a6e5 WatchSource:0}: Error finding container 0b9af210d5426b0b53c7928002b1f9d2df4e3dc6f8257920ba891ffca4c2a6e5: Status 404 returned error can't find the container with id 0b9af210d5426b0b53c7928002b1f9d2df4e3dc6f8257920ba891ffca4c2a6e5 Jul 24 17:42:58.745376 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:58.745351 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cab09e2_baba_4a01_a58f_53c91600107f.slice/crio-e511c57fcf4abff22c591cfcc8947bd776b4a73973f3080b67b79f3b7d5aeb58 WatchSource:0}: Error finding container e511c57fcf4abff22c591cfcc8947bd776b4a73973f3080b67b79f3b7d5aeb58: Status 404 returned error can't find the container with id e511c57fcf4abff22c591cfcc8947bd776b4a73973f3080b67b79f3b7d5aeb58 Jul 24 17:42:58.766967 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:58.766934 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bcdff46_acf9_4858_aee1_d92f3219d7e9.slice/crio-646d562d2f580a180449a7048f7c9aed8df4e96db2c8d224e88737b95b2b4d84 WatchSource:0}: Error finding container 646d562d2f580a180449a7048f7c9aed8df4e96db2c8d224e88737b95b2b4d84: Status 404 returned error can't find the container with id 646d562d2f580a180449a7048f7c9aed8df4e96db2c8d224e88737b95b2b4d84 Jul 24 17:42:58.769660 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:42:58.769621 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb00377e3_0fe2_4fe7_89cb_c892e233959f.slice/crio-bdbe19236fddc44d54402acab4824209f25ff344deba7d4ccbaf27dbd301ffe0 WatchSource:0}: Error finding container bdbe19236fddc44d54402acab4824209f25ff344deba7d4ccbaf27dbd301ffe0: Status 404 returned error can't find the container with id bdbe19236fddc44d54402acab4824209f25ff344deba7d4ccbaf27dbd301ffe0 Jul 24 17:42:59.010630 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.010469 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:59.010779 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.010633 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:42:59.010779 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.010705 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs podName:1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:01.010689758 +0000 UTC m=+6.050483792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs") pod "network-metrics-daemon-5pvds" (UID: "1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:42:59.111599 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.111560 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:59.111743 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.111671 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:42:59.111743 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.111725 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret podName:b88c2b38-14f5-4ef4-8e28-76663ce9e34f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:01.11171288 +0000 UTC m=+6.151506902 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret") pod "global-pull-secret-syncer-zh5f7" (UID: "b88c2b38-14f5-4ef4-8e28-76663ce9e34f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:42:59.172891 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.172864 2559 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 24 17:42:59.212112 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.212068 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:42:59.212307 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.212253 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:42:59.212307 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.212280 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:42:59.212307 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.212293 2559 projected.go:196] Error preparing data for projected volume kube-api-access-dzcln for pod openshift-network-diagnostics/network-check-target-26mkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:42:59.212460 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.212368 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln podName:9aeff889-3a93-4ecb-a76d-925dfa85e4fa nodeName:}" failed. No retries permitted until 2026-07-24 17:43:01.212345394 +0000 UTC m=+6.252139437 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzcln" (UniqueName: "kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln") pod "network-check-target-26mkw" (UID: "9aeff889-3a93-4ecb-a76d-925dfa85e4fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:42:59.441043 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.440967 2559 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 17:37:57 +0000 UTC" deadline="2028-03-27 08:12:35.850577019 +0000 UTC" Jul 24 17:42:59.441043 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.441010 2559 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14678h29m36.409572165s" Jul 24 17:42:59.491966 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.491024 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:42:59.491966 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.491156 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:42:59.491966 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.491641 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:42:59.491966 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.491750 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:42:59.491966 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.491815 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:42:59.491966 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:42:59.491911 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:42:59.521766 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.521730 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fsf7z" event={"ID":"b00377e3-0fe2-4fe7-89cb-c892e233959f","Type":"ContainerStarted","Data":"bdbe19236fddc44d54402acab4824209f25ff344deba7d4ccbaf27dbd301ffe0"} Jul 24 17:42:59.525688 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.525617 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlgtf" event={"ID":"73a09260-b421-4b83-8e67-4c52d34fd50c","Type":"ContainerStarted","Data":"d8012009c6d9df2fadd9cb3b56d1b007d1b44cfc280c4d8c446cb41a42092cc3"} Jul 24 17:42:59.527627 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.527595 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" event={"ID":"3236bb71-d19a-401c-bafc-fbb06ed145fe","Type":"ContainerStarted","Data":"c13613ea77d4f73ea9e912ecd086558edb1beaa53d806634f4d96c289dbf9059"} Jul 24 17:42:59.531428 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.531395 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29ff9" event={"ID":"8bcdff46-acf9-4858-aee1-d92f3219d7e9","Type":"ContainerStarted","Data":"646d562d2f580a180449a7048f7c9aed8df4e96db2c8d224e88737b95b2b4d84"} Jul 24 17:42:59.534434 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.534406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerStarted","Data":"e511c57fcf4abff22c591cfcc8947bd776b4a73973f3080b67b79f3b7d5aeb58"} Jul 24 17:42:59.537269 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.537242 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gnjpf" event={"ID":"3076bbf1-a934-4c5d-a679-a3aa1c824daf","Type":"ContainerStarted","Data":"244ae8aee20a908dd1ba0efe71e954ebe3e1055a406bfadb9c10983f1d311798"} Jul 24 17:42:59.542984 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.542955 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"0b9af210d5426b0b53c7928002b1f9d2df4e3dc6f8257920ba891ffca4c2a6e5"} Jul 24 17:42:59.545626 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.545599 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkd24" event={"ID":"c63c5161-5c4c-44e8-991b-603b5c3b24b9","Type":"ContainerStarted","Data":"1da49a7792aab26871db35c86ce7825cafff80cc7c3bdbd0c12fa36ec3e605b4"} Jul 24 17:42:59.549391 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.549366 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" event={"ID":"96cdd753-0f32-40fa-b472-2dfd5864c909","Type":"ContainerStarted","Data":"a1f03c3d2b1d88939993b9282959bfc3922195890efd62af181bd3c3f5b1c502"} Jul 24 17:42:59.553327 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.553296 2559 generic.go:356] "Generic (PLEG): container finished" podID="25d7561822a46a6ad06fa6a75d9872a7" containerID="dce65507d37fa9c242b0ec823fc4425538aeadda71db3c8ee8fc979be7e55843" exitCode=0 Jul 24 17:42:59.553421 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.553380 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" event={"ID":"25d7561822a46a6ad06fa6a75d9872a7","Type":"ContainerDied","Data":"dce65507d37fa9c242b0ec823fc4425538aeadda71db3c8ee8fc979be7e55843"} Jul 24 17:42:59.558226 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.558197 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" event={"ID":"d2a76f45062ee97c58334419b354384f","Type":"ContainerStarted","Data":"65a3f1fa01d7fd62903596866591c93d62b1738534c9b9ab384e2290c86ff140"} Jul 24 17:42:59.589504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:42:59.589040 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-191.ec2.internal" podStartSLOduration=3.58901688 podStartE2EDuration="3.58901688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:42:59.585632806 +0000 UTC m=+4.625426853" watchObservedRunningTime="2026-07-24 17:42:59.58901688 +0000 UTC m=+4.628810927" Jul 24 17:43:00.582437 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:00.581972 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" event={"ID":"25d7561822a46a6ad06fa6a75d9872a7","Type":"ContainerStarted","Data":"beb1a60daf41a9f4a5bbec841d4d2575b804001cf92903442282b17316f35d2c"} Jul 24 17:43:01.028535 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:01.028450 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:01.028721 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.028625 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:43:01.028721 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.028693 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs podName:1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:05.02867397 +0000 UTC m=+10.068468006 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs") pod "network-metrics-daemon-5pvds" (UID: "1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:43:01.129222 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:01.129179 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:01.129412 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.129388 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:43:01.129474 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.129459 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret podName:b88c2b38-14f5-4ef4-8e28-76663ce9e34f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:05.129437969 +0000 UTC m=+10.169231992 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret") pod "global-pull-secret-syncer-zh5f7" (UID: "b88c2b38-14f5-4ef4-8e28-76663ce9e34f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:43:01.230208 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:01.230156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:01.230424 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.230400 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:43:01.230513 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.230425 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:43:01.230513 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.230439 2559 projected.go:196] Error preparing data for projected volume kube-api-access-dzcln for pod openshift-network-diagnostics/network-check-target-26mkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:43:01.230513 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.230501 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln podName:9aeff889-3a93-4ecb-a76d-925dfa85e4fa nodeName:}" failed. No retries permitted until 2026-07-24 17:43:05.230482287 +0000 UTC m=+10.270276313 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzcln" (UniqueName: "kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln") pod "network-check-target-26mkw" (UID: "9aeff889-3a93-4ecb-a76d-925dfa85e4fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:43:01.491268 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:01.491213 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:01.491689 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.491369 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:01.492126 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:01.491982 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:01.492226 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.492162 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:01.492286 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:01.492242 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:01.492644 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:01.492337 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:03.488914 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:03.488880 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:03.489383 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:03.489008 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:03.489451 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:03.489408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:03.489551 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:03.489529 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:03.491862 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:03.491837 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:03.491970 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:03.491949 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:05.064333 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:05.064299 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:05.064801 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.064497 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:43:05.064801 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.064558 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs podName:1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:13.06453833 +0000 UTC m=+18.104332352 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs") pod "network-metrics-daemon-5pvds" (UID: "1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:43:05.165709 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:05.165664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:05.165910 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.165816 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:43:05.165910 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.165863 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret podName:b88c2b38-14f5-4ef4-8e28-76663ce9e34f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:13.165850541 +0000 UTC m=+18.205644560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret") pod "global-pull-secret-syncer-zh5f7" (UID: "b88c2b38-14f5-4ef4-8e28-76663ce9e34f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:43:05.266286 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:05.266239 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:05.266497 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.266465 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:43:05.266497 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.266491 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:43:05.266636 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.266505 2559 projected.go:196] Error preparing data for projected volume kube-api-access-dzcln for pod openshift-network-diagnostics/network-check-target-26mkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:43:05.266636 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.266567 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln podName:9aeff889-3a93-4ecb-a76d-925dfa85e4fa nodeName:}" failed. No retries permitted until 2026-07-24 17:43:13.266547175 +0000 UTC m=+18.306341199 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzcln" (UniqueName: "kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln") pod "network-check-target-26mkw" (UID: "9aeff889-3a93-4ecb-a76d-925dfa85e4fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:43:05.491217 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:05.491134 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:05.491383 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:05.491134 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:05.491383 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.491258 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:05.491383 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:05.491135 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:05.491383 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.491347 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:05.491592 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:05.491390 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:07.488739 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:07.488706 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:07.489142 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:07.488859 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:07.489207 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:07.489186 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:07.489320 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:07.489299 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:07.489395 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:07.489380 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:07.489521 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:07.489503 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:09.491527 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:09.491492 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:09.491996 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:09.491706 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:09.493676 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:09.492197 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:09.493676 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:09.492336 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:09.493676 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:09.492399 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:09.493676 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:09.492485 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:11.488102 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:11.488072 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:11.488636 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:11.488072 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:11.488636 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:11.488189 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:11.488636 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:11.488077 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:11.488636 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:11.488273 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:11.488636 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:11.488362 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:13.126481 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:13.126447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:13.127008 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.126580 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:43:13.127008 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.126641 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs podName:1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:29.126620376 +0000 UTC m=+34.166414412 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs") pod "network-metrics-daemon-5pvds" (UID: "1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:43:13.227123 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:13.227074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:13.227268 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.227227 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:43:13.227378 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.227300 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret podName:b88c2b38-14f5-4ef4-8e28-76663ce9e34f nodeName:}" failed. No retries permitted until 2026-07-24 17:43:29.22727962 +0000 UTC m=+34.267073640 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret") pod "global-pull-secret-syncer-zh5f7" (UID: "b88c2b38-14f5-4ef4-8e28-76663ce9e34f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:43:13.328097 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:13.328066 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:13.328258 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.328229 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:43:13.328258 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.328249 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:43:13.328373 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.328261 2559 projected.go:196] Error preparing data for projected volume kube-api-access-dzcln for pod openshift-network-diagnostics/network-check-target-26mkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:43:13.328373 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.328323 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln podName:9aeff889-3a93-4ecb-a76d-925dfa85e4fa nodeName:}" failed. No retries permitted until 2026-07-24 17:43:29.328304535 +0000 UTC m=+34.368098570 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzcln" (UniqueName: "kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln") pod "network-check-target-26mkw" (UID: "9aeff889-3a93-4ecb-a76d-925dfa85e4fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:43:13.488239 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:13.488158 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:13.488388 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:13.488158 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:13.488388 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.488286 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:13.488388 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:13.488158 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:13.488388 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.488372 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:13.488624 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:13.488439 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:15.489103 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:15.489070 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:15.489537 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:15.489181 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:15.489599 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:15.489556 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:15.489639 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:15.489621 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:15.489691 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:15.489679 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:15.489754 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:15.489740 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:17.488527 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:17.488487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:17.488527 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:17.488522 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:17.489195 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:17.488487 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:17.489195 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:17.488620 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:17.489195 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:17.488728 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:17.489195 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:17.488832 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:18.632656 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.632221 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-29ff9" event={"ID":"8bcdff46-acf9-4858-aee1-d92f3219d7e9","Type":"ContainerStarted","Data":"a309ca91d5cdb9152f718e40de30572f8dced3b2ab6e289bf55f89c33b5c9431"} Jul 24 17:43:18.633874 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.633847 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cab09e2-baba-4a01-a58f-53c91600107f" containerID="c56a4f9bf5ae3fcfb65afc5ec1a3b68e4808590cf07a009b5172a53ab804ed82" exitCode=0 Jul 24 17:43:18.634007 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.633926 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerDied","Data":"c56a4f9bf5ae3fcfb65afc5ec1a3b68e4808590cf07a009b5172a53ab804ed82"} Jul 24 17:43:18.635325 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.635306 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gnjpf" event={"ID":"3076bbf1-a934-4c5d-a679-a3aa1c824daf","Type":"ContainerStarted","Data":"a163ad59a7684b0017546c7479e8a96382ede7227ab7cb22a20070c35b4d41c7"} Jul 24 17:43:18.638461 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.638442 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"34f4fc602e138c5c0b42bc8dacb614619992dcea4f3b545f41d914a935fef5da"} Jul 24 17:43:18.638565 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.638502 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"a3b3314b7ef2d01ed211895201f91447a542275a4b2971972ca8156a14ecd299"} Jul 24 17:43:18.638613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.638585 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"47c65edaf45bcd6a068492564d5c242b748180cfe524244190a94a77e25888e4"} Jul 24 17:43:18.638661 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.638629 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"622370dd114cb9b831a4d8dcef899721f3db5e0319b944aa29ff106385af591b"} Jul 24 17:43:18.638712 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.638668 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"78d91b5f5f0fdefc949d1525a7a606d4eff078da3d4e60ea44a971b1bed90517"} Jul 24 17:43:18.638712 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.638695 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"b2d060bedfd4475fab2dcceaf41aeb5dad7b1b82e912a07ff85d0e11a9d28329"} Jul 24 17:43:18.639824 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.639803 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkd24" event={"ID":"c63c5161-5c4c-44e8-991b-603b5c3b24b9","Type":"ContainerStarted","Data":"ad26dc94cf719231f86acf3beb64ac1e27ad24284003db9b52eb267cde6ea9e8"} Jul 24 17:43:18.641133 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.641104 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" event={"ID":"96cdd753-0f32-40fa-b472-2dfd5864c909","Type":"ContainerStarted","Data":"25f9178c649789ef7e50bcd0d143d7418c9cdd3b084d422488d8e8e39d3fcd48"} Jul 24 17:43:18.642468 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.642449 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fsf7z" event={"ID":"b00377e3-0fe2-4fe7-89cb-c892e233959f","Type":"ContainerStarted","Data":"22e5fe8227f2a55bff6ee4a1e14b62a261bbe2626b6b3f0eb75707bc6510bbc0"} Jul 24 17:43:18.643693 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.643675 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" event={"ID":"3236bb71-d19a-401c-bafc-fbb06ed145fe","Type":"ContainerStarted","Data":"f8e1f6e52be349e4ab49aaf224331fa376ed6866b702f292589d05834f8d120c"} Jul 24 17:43:18.651402 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.651365 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-29ff9" podStartSLOduration=3.377464695 podStartE2EDuration="22.651353779s" totalImagesPullingTime="19.273889084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:18.651141695 +0000 UTC m=+23.690935736" watchObservedRunningTime="2026-07-24 17:43:18.651353779 +0000 UTC m=+23.691147820" Jul 24 17:43:18.651645 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.651625 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-191.ec2.internal" podStartSLOduration=22.651619675 podStartE2EDuration="22.651619675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:00.600553335 +0000 UTC m=+5.640347381" watchObservedRunningTime="2026-07-24 17:43:18.651619675 +0000 UTC m=+23.691413715" Jul 24 17:43:18.684374 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.682348 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zkd24" podStartSLOduration=3.447510438 podStartE2EDuration="22.682330212s" totalImagesPullingTime="19.234819774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:18.66597423 +0000 UTC m=+23.705768270" watchObservedRunningTime="2026-07-24 17:43:18.682330212 +0000 UTC m=+23.722124254" Jul 24 17:43:18.710459 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.710416 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fsf7z" podStartSLOduration=3.505549518 podStartE2EDuration="22.710401003s" totalImagesPullingTime="19.204851485s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:18.688098063 +0000 UTC m=+23.727892109" watchObservedRunningTime="2026-07-24 17:43:18.710401003 +0000 UTC m=+23.750195044" Jul 24 17:43:18.733488 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.733379 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-mjdtk" podStartSLOduration=4.492310403 podStartE2EDuration="23.733361936s" totalImagesPullingTime="19.241051533s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:18.733026939 +0000 UTC m=+23.772820981" watchObservedRunningTime="2026-07-24 17:43:18.733361936 +0000 UTC m=+23.773155977" Jul 24 17:43:18.753780 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:18.753738 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gnjpf" podStartSLOduration=3.528405213 podStartE2EDuration="22.75372533s" totalImagesPullingTime="19.225320117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:18.753602601 +0000 UTC m=+23.793396638" watchObservedRunningTime="2026-07-24 17:43:18.75372533 +0000 UTC m=+23.793519371" Jul 24 17:43:19.489321 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:19.488888 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:19.489321 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:19.488904 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:19.489321 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:19.489019 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:19.489321 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:19.489063 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:19.489321 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:19.489228 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:19.489321 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:19.489312 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:19.576518 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:19.576415 2559 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 24 17:43:19.647473 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:19.647441 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tlgtf" event={"ID":"73a09260-b421-4b83-8e67-4c52d34fd50c","Type":"ContainerStarted","Data":"24521302590ce8b94b5c37bc122b07d4e87caf300396dcb1f988cfc35e660d2e"} Jul 24 17:43:19.649542 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:19.649511 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" event={"ID":"3236bb71-d19a-401c-bafc-fbb06ed145fe","Type":"ContainerStarted","Data":"92587b3e62c11b392aab924a974a7fadd20b0ed7f21f456936acfcbf0cebd254"} Jul 24 17:43:19.662549 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:19.662464 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tlgtf" podStartSLOduration=4.446171128 podStartE2EDuration="23.662445158s" totalImagesPullingTime="19.21627403s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:19.662215622 +0000 UTC m=+24.702009664" watchObservedRunningTime="2026-07-24 17:43:19.662445158 +0000 UTC m=+24.702239199" Jul 24 17:43:20.460609 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:20.460500 2559 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T17:43:19.576510192Z","UUID":"a11e0c2a-63a8-43fd-98d6-8ec88f2813e9","Handler":null,"Name":"","Endpoint":""} Jul 24 17:43:20.462134 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:20.462112 2559 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 24 17:43:20.462276 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:20.462149 2559 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 24 17:43:21.488596 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:21.488567 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:21.489057 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:21.488572 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:21.489057 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:21.488671 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:21.489057 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:21.488572 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:21.489057 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:21.488745 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:21.489057 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:21.488881 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:21.664587 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:21.664543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" event={"ID":"3236bb71-d19a-401c-bafc-fbb06ed145fe","Type":"ContainerStarted","Data":"9aa2ef9d84c9bc2c83ca0cdf9bdc95e9fbda6f3c8a1657b93c65ca415a878d19"} Jul 24 17:43:21.668389 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:21.668336 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"4334fe88c13e379aa12b3495ab2ab93ae4ae0f1f550d117541ff5247dd6ae1d6"} Jul 24 17:43:21.682431 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:21.682379 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-l9zr4" podStartSLOduration=4.100790396 podStartE2EDuration="25.682360566s" totalImagesPullingTime="21.58157017s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:21.682126082 +0000 UTC m=+26.721920125" watchObservedRunningTime="2026-07-24 17:43:21.682360566 +0000 UTC m=+26.722154609" Jul 24 17:43:23.079490 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.079283 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:43:23.080019 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.079993 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:43:23.488528 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.488456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:23.488675 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.488456 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:23.488675 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.488465 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:23.488755 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:23.488679 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:23.488755 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:23.488557 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:23.488871 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:23.488748 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:23.677825 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.677709 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" event={"ID":"0243df16-ec23-48c9-9451-34655a6cee2b","Type":"ContainerStarted","Data":"62e314d51bee38a50a5f02756f4373320bd1915e69e55656880cae6fabd42320"} Jul 24 17:43:23.678059 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.678041 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:43:23.678720 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.678688 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zkd24" Jul 24 17:43:23.706832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:23.706752 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" podStartSLOduration=8.430231086 podStartE2EDuration="27.706737576s" totalImagesPullingTime="19.27650649s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:23.705552807 +0000 UTC m=+28.745346853" watchObservedRunningTime="2026-07-24 17:43:23.706737576 +0000 UTC m=+28.746531616" Jul 24 17:43:24.681482 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:24.681291 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cab09e2-baba-4a01-a58f-53c91600107f" containerID="740d655a8db979bea7669492dfc7b097a2efe717685e487c36411d3c1e0b70c8" exitCode=0 Jul 24 17:43:24.682393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:24.681377 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerDied","Data":"740d655a8db979bea7669492dfc7b097a2efe717685e487c36411d3c1e0b70c8"} Jul 24 17:43:24.682393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:24.682094 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:43:24.682393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:24.682181 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:43:24.682393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:24.682231 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:43:24.698656 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:24.698628 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:43:24.698806 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:24.698751 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:43:25.492660 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.492345 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:25.492660 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:25.492519 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:25.492660 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.492594 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:25.492959 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:25.492746 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:25.493139 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.493120 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:25.494089 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:25.494057 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:25.810314 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.810150 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zh5f7"] Jul 24 17:43:25.810314 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.810275 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:25.810711 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:25.810390 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:25.814025 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.813997 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26mkw"] Jul 24 17:43:25.814160 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.814103 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:25.814212 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:25.814178 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:25.814621 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.814599 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5pvds"] Jul 24 17:43:25.814721 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:25.814681 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:25.814834 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:25.814809 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:26.687374 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:26.687344 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cab09e2-baba-4a01-a58f-53c91600107f" containerID="d10f89174ae5f71a801103acee143fbbc7292c4edf2cbe21b3bc8dbc596697a7" exitCode=0 Jul 24 17:43:26.687572 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:26.687428 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerDied","Data":"d10f89174ae5f71a801103acee143fbbc7292c4edf2cbe21b3bc8dbc596697a7"} Jul 24 17:43:27.488088 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:27.488057 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:27.488514 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:27.488069 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:27.488514 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:27.488193 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:27.488514 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:27.488068 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:27.488514 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:27.488254 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:27.488514 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:27.488330 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:27.695614 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:27.695391 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerStarted","Data":"bfc51bba515fd4f2671f90e0383a3c7020672f2d727c03fdd2a00b77d60c5e69"} Jul 24 17:43:28.700371 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:28.700334 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cab09e2-baba-4a01-a58f-53c91600107f" containerID="bfc51bba515fd4f2671f90e0383a3c7020672f2d727c03fdd2a00b77d60c5e69" exitCode=0 Jul 24 17:43:28.700856 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:28.700406 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerDied","Data":"bfc51bba515fd4f2671f90e0383a3c7020672f2d727c03fdd2a00b77d60c5e69"} Jul 24 17:43:29.141190 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:29.141156 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:29.141365 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.141261 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:43:29.141365 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.141316 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs podName:1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a nodeName:}" failed. No retries permitted until 2026-07-24 17:44:01.141302636 +0000 UTC m=+66.181096655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs") pod "network-metrics-daemon-5pvds" (UID: "1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 24 17:43:29.242239 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:29.242202 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:29.242406 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.242310 2559 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 24 17:43:29.242406 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.242364 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret podName:b88c2b38-14f5-4ef4-8e28-76663ce9e34f nodeName:}" failed. No retries permitted until 2026-07-24 17:44:01.24235017 +0000 UTC m=+66.282144190 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret") pod "global-pull-secret-syncer-zh5f7" (UID: "b88c2b38-14f5-4ef4-8e28-76663ce9e34f") : object "kube-system"/"original-pull-secret" not registered Jul 24 17:43:29.343403 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:29.343367 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:29.343569 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.343520 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 24 17:43:29.343569 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.343536 2559 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 24 17:43:29.343569 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.343545 2559 projected.go:196] Error preparing data for projected volume kube-api-access-dzcln for pod openshift-network-diagnostics/network-check-target-26mkw: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:43:29.343692 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.343594 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln podName:9aeff889-3a93-4ecb-a76d-925dfa85e4fa nodeName:}" failed. No retries permitted until 2026-07-24 17:44:01.343579677 +0000 UTC m=+66.383373697 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dzcln" (UniqueName: "kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln") pod "network-check-target-26mkw" (UID: "9aeff889-3a93-4ecb-a76d-925dfa85e4fa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 24 17:43:29.488827 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:29.488776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:29.488979 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:29.488776 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:29.488979 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.488926 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:29.488979 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:29.488942 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:29.489280 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.489039 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:29.489280 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:29.489119 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:31.488560 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.488523 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:31.488560 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.488523 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:31.488560 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.488545 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:31.489029 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:31.488663 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-26mkw" podUID="9aeff889-3a93-4ecb-a76d-925dfa85e4fa" Jul 24 17:43:31.489029 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:31.488698 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zh5f7" podUID="b88c2b38-14f5-4ef4-8e28-76663ce9e34f" Jul 24 17:43:31.489029 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:31.488803 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-5pvds" podUID="1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a" Jul 24 17:43:31.820765 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.820730 2559 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-191.ec2.internal" event="NodeReady" Jul 24 17:43:31.820952 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.820892 2559 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 24 17:43:31.887582 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.887546 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-658f8b44f6-hkqlq"] Jul 24 17:43:31.891338 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.891319 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:31.896128 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.896099 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 24 17:43:31.896263 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.896176 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 24 17:43:31.896263 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.896185 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 24 17:43:31.896378 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.896327 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vh2n6\"" Jul 24 17:43:31.905051 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.905032 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 24 17:43:31.915269 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.915032 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-658f8b44f6-hkqlq"] Jul 24 17:43:31.932735 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.932703 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt"] Jul 24 17:43:31.936387 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.936360 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:31.939809 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.939756 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5885fdb759-k2526"] Jul 24 17:43:31.942762 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.942545 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 24 17:43:31.942762 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.942566 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 24 17:43:31.942762 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.942681 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:43:31.943015 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.942849 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk"] Jul 24 17:43:31.943195 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.943100 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:31.943423 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.943403 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 24 17:43:31.944270 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.943639 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-55jwd\"" Jul 24 17:43:31.946452 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.946436 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg"] Jul 24 17:43:31.946573 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.946559 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:31.951106 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.950839 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 24 17:43:31.951106 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.950916 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 24 17:43:31.951106 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.950924 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4z74v\"" Jul 24 17:43:31.951106 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.950841 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 24 17:43:31.951106 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.951011 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 24 17:43:31.952910 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.952889 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-cbm78"] Jul 24 17:43:31.953046 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.953026 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg" Jul 24 17:43:31.953468 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.953428 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 24 17:43:31.956590 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.956572 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-dz2pl"] Jul 24 17:43:31.956711 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.956696 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:31.959594 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.959576 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf"] Jul 24 17:43:31.959734 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.959709 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:31.963054 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.963037 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:31.974626 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.974606 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hddhl\"" Jul 24 17:43:31.975110 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.975075 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 24 17:43:31.975227 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.975188 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 24 17:43:31.975661 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.975645 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 24 17:43:31.976045 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.976026 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:43:31.976417 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.976399 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 24 17:43:31.976504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.976448 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 24 17:43:31.976668 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.976647 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-scjxw\"" Jul 24 17:43:31.978837 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.978817 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f"] Jul 24 17:43:31.980907 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.980871 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 24 17:43:31.981153 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.981133 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 24 17:43:31.981255 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.981152 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 24 17:43:31.981255 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.981218 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 24 17:43:31.981366 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.981275 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 24 17:43:31.981366 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.981318 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:43:31.981907 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.981881 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9n4kl\"" Jul 24 17:43:31.982000 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.981935 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:43:31.982061 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.982018 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5kpm4\"" Jul 24 17:43:31.982371 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.981880 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 24 17:43:31.982371 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.982149 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-kdccn\"" Jul 24 17:43:31.982371 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.982223 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 24 17:43:31.982371 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.982264 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-758d7d5dd5-wq9dp"] Jul 24 17:43:31.982599 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.982420 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:31.982599 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.982066 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 24 17:43:31.986070 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.986054 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-k2526"] Jul 24 17:43:31.986760 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.986741 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-dz2pl"] Jul 24 17:43:31.986863 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.986777 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt"] Jul 24 17:43:31.986863 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.986503 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:31.987485 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.987443 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cbm78"] Jul 24 17:43:31.990840 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.990822 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:43:31.996747 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.996721 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 24 17:43:31.996947 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.996927 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dwmxq\"" Jul 24 17:43:31.997051 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.997035 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 24 17:43:31.997158 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.997083 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 24 17:43:31.997361 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.997343 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 24 17:43:31.998021 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.997853 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 24 17:43:31.999630 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:31.999615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 24 17:43:32.000038 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.000017 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 24 17:43:32.000147 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.000130 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sxl67\"" Jul 24 17:43:32.000402 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.000378 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 24 17:43:32.001197 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.001178 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk"] Jul 24 17:43:32.007836 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.006422 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 24 17:43:32.035949 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.035919 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg"] Jul 24 17:43:32.038863 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.038837 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f"] Jul 24 17:43:32.039037 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.038902 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-758d7d5dd5-wq9dp"] Jul 24 17:43:32.039333 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.039274 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-26s9g"] Jul 24 17:43:32.043000 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.042979 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf"] Jul 24 17:43:32.043095 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.043012 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b"] Jul 24 17:43:32.043158 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.043121 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:32.046238 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.046217 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" Jul 24 17:43:32.046548 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.046534 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hnvlj"] Jul 24 17:43:32.047216 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.047194 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 24 17:43:32.047753 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.047735 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-982db\"" Jul 24 17:43:32.049659 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.049642 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45"] Jul 24 17:43:32.049753 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.049662 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 24 17:43:32.049852 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.049763 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hnvlj" Jul 24 17:43:32.052835 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.052765 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g"] Jul 24 17:43:32.052968 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.052932 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.053047 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.052984 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 24 17:43:32.053214 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.053194 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 24 17:43:32.053397 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.053376 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 24 17:43:32.053486 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.053406 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 24 17:43:32.053737 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.053716 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 24 17:43:32.054819 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.054777 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 24 17:43:32.054926 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.054907 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 24 17:43:32.055053 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.055037 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t2cd6\"" Jul 24 17:43:32.055499 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.055470 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-42xx7\"" Jul 24 17:43:32.055910 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.055896 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp"] Jul 24 17:43:32.056036 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.056021 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.056419 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.056399 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 24 17:43:32.057423 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.057402 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 24 17:43:32.057502 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.057409 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 24 17:43:32.058164 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.057672 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 24 17:43:32.058936 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.058918 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp" Jul 24 17:43:32.062658 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r52fs\" (UniqueName: \"kubernetes.io/projected/b9c8447a-8f08-4d76-a7d3-103f60b7e495-kube-api-access-r52fs\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.062738 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062666 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/02e6ee41-b12e-42ee-9589-f0435e5ae956-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.062738 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062688 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9c8447a-8f08-4d76-a7d3-103f60b7e495-snapshots\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.062738 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062710 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:32.062738 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062735 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-bound-sa-token\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.062940 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062772 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c8447a-8f08-4d76-a7d3-103f60b7e495-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.062940 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062815 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9c8447a-8f08-4d76-a7d3-103f60b7e495-serving-cert\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.062940 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062835 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d642fa46-5c47-4435-955b-1f73afb4fb81-tmp-dir\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.062940 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062852 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5zvx\" (UniqueName: \"kubernetes.io/projected/72416b1c-f944-4c29-b8fe-5a3b053825f7-kube-api-access-w5zvx\") pod \"volume-data-source-validator-549c6d54cf-wqnfg\" (UID: \"72416b1c-f944-4c29-b8fe-5a3b053825f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg" Jul 24 17:43:32.062940 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062867 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c64451b6-8365-4a7a-8512-baaee772473d-config\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.062940 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d642fa46-5c47-4435-955b-1f73afb4fb81-config-volume\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.062940 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c8447a-8f08-4d76-a7d3-103f60b7e495-tmp\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.062986 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wk5g2\" (UniqueName: \"kubernetes.io/projected/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-kube-api-access-wk5g2\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c64451b6-8365-4a7a-8512-baaee772473d-serving-cert\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063038 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063063 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-image-registry-private-configuration\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063083 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fp94\" (UniqueName: \"kubernetes.io/projected/d642fa46-5c47-4435-955b-1f73afb4fb81-kube-api-access-7fp94\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtmsv\" (UniqueName: \"kubernetes.io/projected/c64451b6-8365-4a7a-8512-baaee772473d-kube-api-access-dtmsv\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.063211 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063148 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cqbf\" (UniqueName: \"kubernetes.io/projected/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-kube-api-access-9cqbf\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063214 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063252 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6vfh\" (UniqueName: \"kubernetes.io/projected/02e6ee41-b12e-42ee-9589-f0435e5ae956-kube-api-access-x6vfh\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063278 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4r94\" (UniqueName: \"kubernetes.io/projected/aebd90e8-411f-4c87-bed7-eb3a1ca82458-kube-api-access-m4r94\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063309 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4clh2\" (UniqueName: \"kubernetes.io/projected/ed3b05b8-f53f-4148-8b50-dac538f09ad6-kube-api-access-4clh2\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063341 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa94510b-f873-4c29-8b7b-668248d6e4cb-ca-trust-extracted\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063370 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aebd90e8-411f-4c87-bed7-eb3a1ca82458-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063394 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063438 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj448\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-kube-api-access-sj448\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.063515 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063475 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-default-certificate\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063495 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aebd90e8-411f-4c87-bed7-eb3a1ca82458-config\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063537 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c8447a-8f08-4d76-a7d3-103f60b7e495-service-ca-bundle\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063596 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-stats-auth\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063643 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c64451b6-8365-4a7a-8512-baaee772473d-trusted-ca\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063677 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-installation-pull-secrets\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063701 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-certificates\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.063832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.063739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-trusted-ca\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.064655 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.064634 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 24 17:43:32.064655 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.064652 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 24 17:43:32.064880 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.064864 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-7prg7\"" Jul 24 17:43:32.064923 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.064902 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 24 17:43:32.070967 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.070910 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hnvlj"] Jul 24 17:43:32.078172 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.078150 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-26s9g"] Jul 24 17:43:32.078823 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.078773 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b"] Jul 24 17:43:32.087937 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.087918 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45"] Jul 24 17:43:32.088732 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.088714 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g"] Jul 24 17:43:32.121803 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.121753 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp"] Jul 24 17:43:32.165055 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165020 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.165228 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165070 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97frj\" (UniqueName: \"kubernetes.io/projected/88676e21-31ea-4fdf-963d-04c2e0fb7c75-kube-api-access-97frj\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.165228 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165093 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7h89\" (UniqueName: \"kubernetes.io/projected/a1c0f1b0-af8c-4b28-a678-386655111019-kube-api-access-m7h89\") pod \"managed-serviceaccount-addon-agent-5969db5bc4-2qb6b\" (UID: \"a1c0f1b0-af8c-4b28-a678-386655111019\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" Jul 24 17:43:32.165228 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165127 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6vfh\" (UniqueName: \"kubernetes.io/projected/02e6ee41-b12e-42ee-9589-f0435e5ae956-kube-api-access-x6vfh\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.165228 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165182 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4r94\" (UniqueName: \"kubernetes.io/projected/aebd90e8-411f-4c87-bed7-eb3a1ca82458-kube-api-access-m4r94\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.165228 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165218 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edfb93c0-838e-40ee-8fda-324733002b82-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:32.165473 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165377 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.165473 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4clh2\" (UniqueName: \"kubernetes.io/projected/ed3b05b8-f53f-4148-8b50-dac538f09ad6-kube-api-access-4clh2\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.165473 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa94510b-f873-4c29-8b7b-668248d6e4cb-ca-trust-extracted\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.165590 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165483 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aebd90e8-411f-4c87-bed7-eb3a1ca82458-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.165590 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165556 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88676e21-31ea-4fdf-963d-04c2e0fb7c75-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.165670 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.165670 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165615 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj448\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-kube-api-access-sj448\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.165670 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165643 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-default-certificate\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.165670 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165666 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aebd90e8-411f-4c87-bed7-eb3a1ca82458-config\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.165849 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165694 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c8447a-8f08-4d76-a7d3-103f60b7e495-service-ca-bundle\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.165849 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-ca\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.165849 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165753 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.165849 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165776 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4fc4\" (UniqueName: \"kubernetes.io/projected/f5e968a2-537a-4759-bc98-e669d89fe627-kube-api-access-q4fc4\") pod \"ingress-canary-hnvlj\" (UID: \"f5e968a2-537a-4759-bc98-e669d89fe627\") " pod="openshift-ingress-canary/ingress-canary-hnvlj" Jul 24 17:43:32.165849 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165839 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-stats-auth\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.166127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165862 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.166127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.165861 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa94510b-f873-4c29-8b7b-668248d6e4cb-ca-trust-extracted\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.166127 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.165982 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:43:32.166127 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.166005 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f8b44f6-hkqlq: secret "image-registry-tls" not found Jul 24 17:43:32.166127 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.166073 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls podName:fa94510b-f873-4c29-8b7b-668248d6e4cb nodeName:}" failed. No retries permitted until 2026-07-24 17:43:32.666051637 +0000 UTC m=+37.705845656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls") pod "image-registry-658f8b44f6-hkqlq" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb") : secret "image-registry-tls" not found Jul 24 17:43:32.166127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.166116 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.166149 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c64451b6-8365-4a7a-8512-baaee772473d-trusted-ca\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.166302 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-installation-pull-secrets\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.166343 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e968a2-537a-4759-bc98-e669d89fe627-cert\") pod \"ingress-canary-hnvlj\" (UID: \"f5e968a2-537a-4759-bc98-e669d89fe627\") " pod="openshift-ingress-canary/ingress-canary-hnvlj" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.166911 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aebd90e8-411f-4c87-bed7-eb3a1ca82458-config\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.166998 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c8447a-8f08-4d76-a7d3-103f60b7e495-service-ca-bundle\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.167072 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.167121 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:32.667104633 +0000 UTC m=+37.706898667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : secret "router-metrics-certs-default" not found Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-certificates\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167348 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-trusted-ca\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a671467a-5ae1-4531-86ec-f36ea72cd2a0-klusterlet-config\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r52fs\" (UniqueName: \"kubernetes.io/projected/b9c8447a-8f08-4d76-a7d3-103f60b7e495-kube-api-access-r52fs\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167423 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167442 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/02e6ee41-b12e-42ee-9589-f0435e5ae956-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167464 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a671467a-5ae1-4531-86ec-f36ea72cd2a0-tmp\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.169504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167491 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc62m\" (UniqueName: \"kubernetes.io/projected/6690c968-3d7f-4ee5-9c7f-9d83510f7727-kube-api-access-sc62m\") pod \"network-check-source-7d88cb55d8-zszlp\" (UID: \"6690c968-3d7f-4ee5-9c7f-9d83510f7727\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167527 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9c8447a-8f08-4d76-a7d3-103f60b7e495-snapshots\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167550 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167576 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-bound-sa-token\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167600 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c8447a-8f08-4d76-a7d3-103f60b7e495-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167627 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9c8447a-8f08-4d76-a7d3-103f60b7e495-serving-cert\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167657 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d642fa46-5c47-4435-955b-1f73afb4fb81-tmp-dir\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167683 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5zvx\" (UniqueName: \"kubernetes.io/projected/72416b1c-f944-4c29-b8fe-5a3b053825f7-kube-api-access-w5zvx\") pod \"volume-data-source-validator-549c6d54cf-wqnfg\" (UID: \"72416b1c-f944-4c29-b8fe-5a3b053825f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167706 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c64451b6-8365-4a7a-8512-baaee772473d-config\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167775 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d642fa46-5c47-4435-955b-1f73afb4fb81-config-volume\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167819 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167845 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c8447a-8f08-4d76-a7d3-103f60b7e495-tmp\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.170212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167984 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-certificates\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.171164 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.167879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wk5g2\" (UniqueName: \"kubernetes.io/projected/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-kube-api-access-wk5g2\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:32.171283 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c64451b6-8365-4a7a-8512-baaee772473d-serving-cert\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.171283 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171234 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1c0f1b0-af8c-4b28-a678-386655111019-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5969db5bc4-2qb6b\" (UID: \"a1c0f1b0-af8c-4b28-a678-386655111019\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" Jul 24 17:43:32.171283 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.171424 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171285 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-image-registry-private-configuration\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.171424 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171305 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfxx\" (UniqueName: \"kubernetes.io/projected/a671467a-5ae1-4531-86ec-f36ea72cd2a0-kube-api-access-tcfxx\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.171424 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171326 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-hub\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.171424 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171353 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.171424 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171373 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fp94\" (UniqueName: \"kubernetes.io/projected/d642fa46-5c47-4435-955b-1f73afb4fb81-kube-api-access-7fp94\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.171424 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171396 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtmsv\" (UniqueName: \"kubernetes.io/projected/c64451b6-8365-4a7a-8512-baaee772473d-kube-api-access-dtmsv\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.171424 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cqbf\" (UniqueName: \"kubernetes.io/projected/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-kube-api-access-9cqbf\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.171714 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171597 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.172018 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.171954 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b9c8447a-8f08-4d76-a7d3-103f60b7e495-tmp\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.172813 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.172770 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-trusted-ca\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.173250 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.173227 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-installation-pull-secrets\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.173490 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.173468 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aebd90e8-411f-4c87-bed7-eb3a1ca82458-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.173835 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.173815 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b9c8447a-8f08-4d76-a7d3-103f60b7e495-trusted-ca-bundle\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.174021 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.174001 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-default-certificate\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.174292 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.174269 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:43:32.174472 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.174457 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls podName:d642fa46-5c47-4435-955b-1f73afb4fb81 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:32.67442989 +0000 UTC m=+37.714223931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls") pod "dns-default-cbm78" (UID: "d642fa46-5c47-4435-955b-1f73afb4fb81") : secret "dns-default-metrics-tls" not found Jul 24 17:43:32.174546 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.174483 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-stats-auth\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.174546 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.174520 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:32.174645 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.174562 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls podName:02e6ee41-b12e-42ee-9589-f0435e5ae956 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:32.674544292 +0000 UTC m=+37.714338311 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-cvpqk" (UID: "02e6ee41-b12e-42ee-9589-f0435e5ae956") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:32.174965 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.174943 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/02e6ee41-b12e-42ee-9589-f0435e5ae956-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.175527 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.175502 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d642fa46-5c47-4435-955b-1f73afb4fb81-tmp-dir\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.175617 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.175542 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:43:32.175677 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.175619 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls podName:96e88041-fc3f-4695-b1fc-6c3bbe62b3a9 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:32.675601318 +0000 UTC m=+37.715395342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls") pod "cluster-samples-operator-6748489656-9xc5f" (UID: "96e88041-fc3f-4695-b1fc-6c3bbe62b3a9") : secret "samples-operator-tls" not found Jul 24 17:43:32.175677 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.175616 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/b9c8447a-8f08-4d76-a7d3-103f60b7e495-snapshots\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.175855 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.168247 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/c64451b6-8365-4a7a-8512-baaee772473d-trusted-ca\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.176124 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.176104 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:32.676087771 +0000 UTC m=+37.715881793 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : configmap references non-existent config key: service-ca.crt Jul 24 17:43:32.176216 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.176195 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d642fa46-5c47-4435-955b-1f73afb4fb81-config-volume\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.176448 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.176424 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c64451b6-8365-4a7a-8512-baaee772473d-config\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.176720 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.176702 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c64451b6-8365-4a7a-8512-baaee772473d-serving-cert\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.177447 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.177411 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-image-registry-private-configuration\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.181083 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.181063 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/b9c8447a-8f08-4d76-a7d3-103f60b7e495-serving-cert\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.198617 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.198593 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cqbf\" (UniqueName: \"kubernetes.io/projected/6f5e6ac9-6056-4018-a77e-33ac5e69cf72-kube-api-access-9cqbf\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf\" (UID: \"6f5e6ac9-6056-4018-a77e-33ac5e69cf72\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.199258 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.199238 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4r94\" (UniqueName: \"kubernetes.io/projected/aebd90e8-411f-4c87-bed7-eb3a1ca82458-kube-api-access-m4r94\") pod \"service-ca-operator-5d6c75f84f-lk7dt\" (UID: \"aebd90e8-411f-4c87-bed7-eb3a1ca82458\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.203262 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.203240 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6vfh\" (UniqueName: \"kubernetes.io/projected/02e6ee41-b12e-42ee-9589-f0435e5ae956-kube-api-access-x6vfh\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.203727 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.203687 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4clh2\" (UniqueName: \"kubernetes.io/projected/ed3b05b8-f53f-4148-8b50-dac538f09ad6-kube-api-access-4clh2\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.203991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.203968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r52fs\" (UniqueName: \"kubernetes.io/projected/b9c8447a-8f08-4d76-a7d3-103f60b7e495-kube-api-access-r52fs\") pod \"insights-operator-5885fdb759-k2526\" (UID: \"b9c8447a-8f08-4d76-a7d3-103f60b7e495\") " pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.204170 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.204150 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5zvx\" (UniqueName: \"kubernetes.io/projected/72416b1c-f944-4c29-b8fe-5a3b053825f7-kube-api-access-w5zvx\") pod \"volume-data-source-validator-549c6d54cf-wqnfg\" (UID: \"72416b1c-f944-4c29-b8fe-5a3b053825f7\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg" Jul 24 17:43:32.211156 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.211134 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wk5g2\" (UniqueName: \"kubernetes.io/projected/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-kube-api-access-wk5g2\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:32.211364 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.211345 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fp94\" (UniqueName: \"kubernetes.io/projected/d642fa46-5c47-4435-955b-1f73afb4fb81-kube-api-access-7fp94\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.211916 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.211900 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj448\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-kube-api-access-sj448\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.212587 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.212566 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-bound-sa-token\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.215280 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.215242 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtmsv\" (UniqueName: \"kubernetes.io/projected/c64451b6-8365-4a7a-8512-baaee772473d-kube-api-access-dtmsv\") pod \"console-operator-584dcb6b54-dz2pl\" (UID: \"c64451b6-8365-4a7a-8512-baaee772473d\") " pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.251642 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.251610 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" Jul 24 17:43:32.259190 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.259165 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5885fdb759-k2526" Jul 24 17:43:32.272623 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272590 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7h89\" (UniqueName: \"kubernetes.io/projected/a1c0f1b0-af8c-4b28-a678-386655111019-kube-api-access-m7h89\") pod \"managed-serviceaccount-addon-agent-5969db5bc4-2qb6b\" (UID: \"a1c0f1b0-af8c-4b28-a678-386655111019\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" Jul 24 17:43:32.272749 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272637 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edfb93c0-838e-40ee-8fda-324733002b82-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:32.272749 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272664 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.272749 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272692 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88676e21-31ea-4fdf-963d-04c2e0fb7c75-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.272749 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-ca\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.272957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272797 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4fc4\" (UniqueName: \"kubernetes.io/projected/f5e968a2-537a-4759-bc98-e669d89fe627-kube-api-access-q4fc4\") pod \"ingress-canary-hnvlj\" (UID: \"f5e968a2-537a-4759-bc98-e669d89fe627\") " pod="openshift-ingress-canary/ingress-canary-hnvlj" Jul 24 17:43:32.272957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e968a2-537a-4759-bc98-e669d89fe627-cert\") pod \"ingress-canary-hnvlj\" (UID: \"f5e968a2-537a-4759-bc98-e669d89fe627\") " pod="openshift-ingress-canary/ingress-canary-hnvlj" Jul 24 17:43:32.272957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272888 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a671467a-5ae1-4531-86ec-f36ea72cd2a0-klusterlet-config\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.272957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:32.272957 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272954 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a671467a-5ae1-4531-86ec-f36ea72cd2a0-tmp\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.273119 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.272980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc62m\" (UniqueName: \"kubernetes.io/projected/6690c968-3d7f-4ee5-9c7f-9d83510f7727-kube-api-access-sc62m\") pod \"network-check-source-7d88cb55d8-zszlp\" (UID: \"6690c968-3d7f-4ee5-9c7f-9d83510f7727\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp" Jul 24 17:43:32.273119 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.273119 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273095 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1c0f1b0-af8c-4b28-a678-386655111019-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5969db5bc4-2qb6b\" (UID: \"a1c0f1b0-af8c-4b28-a678-386655111019\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" Jul 24 17:43:32.273215 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273139 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfxx\" (UniqueName: \"kubernetes.io/projected/a671467a-5ae1-4531-86ec-f36ea72cd2a0-kube-api-access-tcfxx\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.273215 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-hub\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.273281 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273219 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97frj\" (UniqueName: \"kubernetes.io/projected/88676e21-31ea-4fdf-963d-04c2e0fb7c75-kube-api-access-97frj\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.273341 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273313 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg" Jul 24 17:43:32.273457 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273369 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/edfb93c0-838e-40ee-8fda-324733002b82-nginx-conf\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:32.273457 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273414 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/88676e21-31ea-4fdf-963d-04c2e0fb7c75-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.273575 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.273480 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 17:43:32.273575 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.273546 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert podName:edfb93c0-838e-40ee-8fda-324733002b82 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:32.773527371 +0000 UTC m=+37.813321392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-26s9g" (UID: "edfb93c0-838e-40ee-8fda-324733002b82") : secret "networking-console-plugin-cert" not found Jul 24 17:43:32.273998 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.273959 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a671467a-5ae1-4531-86ec-f36ea72cd2a0-tmp\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.276292 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.276273 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f5e968a2-537a-4759-bc98-e669d89fe627-cert\") pod \"ingress-canary-hnvlj\" (UID: \"f5e968a2-537a-4759-bc98-e669d89fe627\") " pod="openshift-ingress-canary/ingress-canary-hnvlj" Jul 24 17:43:32.276640 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.276620 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-ca\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.276993 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.276967 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.277084 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.276979 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/a1c0f1b0-af8c-4b28-a678-386655111019-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5969db5bc4-2qb6b\" (UID: \"a1c0f1b0-af8c-4b28-a678-386655111019\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" Jul 24 17:43:32.277166 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.277141 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/a671467a-5ae1-4531-86ec-f36ea72cd2a0-klusterlet-config\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.277721 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.277699 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-hub\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.278724 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.278706 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/88676e21-31ea-4fdf-963d-04c2e0fb7c75-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.285837 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.285814 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7h89\" (UniqueName: \"kubernetes.io/projected/a1c0f1b0-af8c-4b28-a678-386655111019-kube-api-access-m7h89\") pod \"managed-serviceaccount-addon-agent-5969db5bc4-2qb6b\" (UID: \"a1c0f1b0-af8c-4b28-a678-386655111019\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" Jul 24 17:43:32.291241 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.291223 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:32.296964 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.296925 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97frj\" (UniqueName: \"kubernetes.io/projected/88676e21-31ea-4fdf-963d-04c2e0fb7c75-kube-api-access-97frj\") pod \"cluster-proxy-proxy-agent-79b4ffcdcc-hzf45\" (UID: \"88676e21-31ea-4fdf-963d-04c2e0fb7c75\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.297845 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.297818 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" Jul 24 17:43:32.298283 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.298259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfxx\" (UniqueName: \"kubernetes.io/projected/a671467a-5ae1-4531-86ec-f36ea72cd2a0-kube-api-access-tcfxx\") pod \"klusterlet-addon-workmgr-84ff5444c6-h5z2g\" (UID: \"a671467a-5ae1-4531-86ec-f36ea72cd2a0\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.300360 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.300336 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc62m\" (UniqueName: \"kubernetes.io/projected/6690c968-3d7f-4ee5-9c7f-9d83510f7727-kube-api-access-sc62m\") pod \"network-check-source-7d88cb55d8-zszlp\" (UID: \"6690c968-3d7f-4ee5-9c7f-9d83510f7727\") " pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp" Jul 24 17:43:32.301199 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.301181 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4fc4\" (UniqueName: \"kubernetes.io/projected/f5e968a2-537a-4759-bc98-e669d89fe627-kube-api-access-q4fc4\") pod \"ingress-canary-hnvlj\" (UID: \"f5e968a2-537a-4759-bc98-e669d89fe627\") " pod="openshift-ingress-canary/ingress-canary-hnvlj" Jul 24 17:43:32.372373 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.372289 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" Jul 24 17:43:32.381433 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.381410 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hnvlj" Jul 24 17:43:32.396433 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.396399 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:43:32.403224 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.403205 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:32.408936 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.408913 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp" Jul 24 17:43:32.677665 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.677577 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:32.677665 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.677640 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.677737 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.677856 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.677864 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.677873 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f8b44f6-hkqlq: secret "image-registry-tls" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.677866 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.677914 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.677921 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls podName:96e88041-fc3f-4695-b1fc-6c3bbe62b3a9 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:33.677906826 +0000 UTC m=+38.717700845 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls") pod "cluster-samples-operator-6748489656-9xc5f" (UID: "96e88041-fc3f-4695-b1fc-6c3bbe62b3a9") : secret "samples-operator-tls" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.677951 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls podName:fa94510b-f873-4c29-8b7b-668248d6e4cb nodeName:}" failed. No retries permitted until 2026-07-24 17:43:33.677937989 +0000 UTC m=+38.717732011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls") pod "image-registry-658f8b44f6-hkqlq" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb") : secret "image-registry-tls" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.677981 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:33.677964014 +0000 UTC m=+38.717758041 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : secret "router-metrics-certs-default" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.678015 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:33.678005243 +0000 UTC m=+38.717799264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : configmap references non-existent config key: service-ca.crt Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.678060 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.678088 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.678171 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.678193 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:32.678405 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.678196 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls podName:d642fa46-5c47-4435-955b-1f73afb4fb81 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:33.678188948 +0000 UTC m=+38.717982967 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls") pod "dns-default-cbm78" (UID: "d642fa46-5c47-4435-955b-1f73afb4fb81") : secret "dns-default-metrics-tls" not found Jul 24 17:43:32.679230 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.678236 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls podName:02e6ee41-b12e-42ee-9589-f0435e5ae956 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:33.678226706 +0000 UTC m=+38.718020724 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-cvpqk" (UID: "02e6ee41-b12e-42ee-9589-f0435e5ae956") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:32.778840 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:32.778771 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:32.779007 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.778908 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 17:43:32.779007 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:32.778969 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert podName:edfb93c0-838e-40ee-8fda-324733002b82 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:33.77895251 +0000 UTC m=+38.818746531 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-26s9g" (UID: "edfb93c0-838e-40ee-8fda-324733002b82") : secret "networking-console-plugin-cert" not found Jul 24 17:43:33.488944 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.487824 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:43:33.488944 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.488515 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:43:33.489866 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.489579 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:43:33.513213 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.512424 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9txdv\"" Jul 24 17:43:33.513213 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.512672 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 17:43:33.513213 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.512876 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pdhgn\"" Jul 24 17:43:33.513213 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.513071 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 17:43:33.694421 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.694334 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:33.694421 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.694403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:33.695093 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.694484 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:33.695093 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.694536 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:33.695093 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.694570 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:33.695093 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.694608 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:33.695093 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.694760 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:43:33.695093 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.694987 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:43:33.695093 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695054 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls podName:96e88041-fc3f-4695-b1fc-6c3bbe62b3a9 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.695031939 +0000 UTC m=+40.734825979 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls") pod "cluster-samples-operator-6748489656-9xc5f" (UID: "96e88041-fc3f-4695-b1fc-6c3bbe62b3a9") : secret "samples-operator-tls" not found Jul 24 17:43:33.695524 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695479 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:43:33.695524 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695497 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f8b44f6-hkqlq: secret "image-registry-tls" not found Jul 24 17:43:33.695641 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695545 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls podName:fa94510b-f873-4c29-8b7b-668248d6e4cb nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.695530045 +0000 UTC m=+40.735324077 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls") pod "image-registry-658f8b44f6-hkqlq" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb") : secret "image-registry-tls" not found Jul 24 17:43:33.695641 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695604 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:43:33.695641 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695637 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.695626794 +0000 UTC m=+40.735420819 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : secret "router-metrics-certs-default" not found Jul 24 17:43:33.695989 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695701 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.695691824 +0000 UTC m=+40.735485862 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : configmap references non-existent config key: service-ca.crt Jul 24 17:43:33.695989 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695757 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:33.695989 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695813 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls podName:02e6ee41-b12e-42ee-9589-f0435e5ae956 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.695802914 +0000 UTC m=+40.735596936 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-cvpqk" (UID: "02e6ee41-b12e-42ee-9589-f0435e5ae956") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:33.695989 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.695868 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls podName:d642fa46-5c47-4435-955b-1f73afb4fb81 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.695856785 +0000 UTC m=+40.735650804 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls") pod "dns-default-cbm78" (UID: "d642fa46-5c47-4435-955b-1f73afb4fb81") : secret "dns-default-metrics-tls" not found Jul 24 17:43:33.725172 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.724132 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerStarted","Data":"2c71d5d31503f7c6e380091d21431eb103e9a189ac27dc99cc3e94340e5d9ad1"} Jul 24 17:43:33.732867 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.732195 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg"] Jul 24 17:43:33.741937 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.741911 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp"] Jul 24 17:43:33.760699 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.760665 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hnvlj"] Jul 24 17:43:33.760848 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.760737 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g"] Jul 24 17:43:33.760848 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.760763 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45"] Jul 24 17:43:33.775877 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.775848 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b"] Jul 24 17:43:33.790659 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.790625 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf"] Jul 24 17:43:33.794100 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.794076 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt"] Jul 24 17:43:33.795492 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.795466 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:33.795674 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.795656 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 17:43:33.795739 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:33.795729 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert podName:edfb93c0-838e-40ee-8fda-324733002b82 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:35.795713053 +0000 UTC m=+40.835507074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-26s9g" (UID: "edfb93c0-838e-40ee-8fda-324733002b82") : secret "networking-console-plugin-cert" not found Jul 24 17:43:33.815193 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.815171 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-dz2pl"] Jul 24 17:43:33.823824 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:33.823777 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5885fdb759-k2526"] Jul 24 17:43:33.848237 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:43:33.848200 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88676e21_31ea_4fdf_963d_04c2e0fb7c75.slice/crio-57b63b977874b8e9e171ed3e46e8b1706d1b40b7ab5d1a6def9b2c1ffc94bd54 WatchSource:0}: Error finding container 57b63b977874b8e9e171ed3e46e8b1706d1b40b7ab5d1a6def9b2c1ffc94bd54: Status 404 returned error can't find the container with id 57b63b977874b8e9e171ed3e46e8b1706d1b40b7ab5d1a6def9b2c1ffc94bd54 Jul 24 17:43:33.853999 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:43:33.853954 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1c0f1b0_af8c_4b28_a678_386655111019.slice/crio-55b83aaed970287de8c0dd502bbb8d7d0aac2b8561df5ea228d8453dc11d4c1e WatchSource:0}: Error finding container 55b83aaed970287de8c0dd502bbb8d7d0aac2b8561df5ea228d8453dc11d4c1e: Status 404 returned error can't find the container with id 55b83aaed970287de8c0dd502bbb8d7d0aac2b8561df5ea228d8453dc11d4c1e Jul 24 17:43:33.854247 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:43:33.854153 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc64451b6_8365_4a7a_8512_baaee772473d.slice/crio-8526b4426775ab1aac67a0a6a36ee3a54897c3dae595574eed6a9eed220ad94f WatchSource:0}: Error finding container 8526b4426775ab1aac67a0a6a36ee3a54897c3dae595574eed6a9eed220ad94f: Status 404 returned error can't find the container with id 8526b4426775ab1aac67a0a6a36ee3a54897c3dae595574eed6a9eed220ad94f Jul 24 17:43:34.733592 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.733527 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" event={"ID":"88676e21-31ea-4fdf-963d-04c2e0fb7c75","Type":"ContainerStarted","Data":"57b63b977874b8e9e171ed3e46e8b1706d1b40b7ab5d1a6def9b2c1ffc94bd54"} Jul 24 17:43:34.751912 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.751874 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" event={"ID":"a671467a-5ae1-4531-86ec-f36ea72cd2a0","Type":"ContainerStarted","Data":"582097ef13ca73ec527c599520e7a344330cbd6de0f64ff04ebc90617b80b808"} Jul 24 17:43:34.757801 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.757751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-k2526" event={"ID":"b9c8447a-8f08-4d76-a7d3-103f60b7e495","Type":"ContainerStarted","Data":"bd3083703dad1fb9bbea8fdccd7182b92ecc75a55dfed1d195598eeb684ff349"} Jul 24 17:43:34.762745 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.762707 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hnvlj" event={"ID":"f5e968a2-537a-4759-bc98-e669d89fe627","Type":"ContainerStarted","Data":"fc7ad443bd50e2d92c1e0c8b1776462c2684c280820c94749194510b4e9ce0f0"} Jul 24 17:43:34.773264 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.772233 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cab09e2-baba-4a01-a58f-53c91600107f" containerID="2c71d5d31503f7c6e380091d21431eb103e9a189ac27dc99cc3e94340e5d9ad1" exitCode=0 Jul 24 17:43:34.773264 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.772326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerDied","Data":"2c71d5d31503f7c6e380091d21431eb103e9a189ac27dc99cc3e94340e5d9ad1"} Jul 24 17:43:34.780266 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.780236 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" event={"ID":"6f5e6ac9-6056-4018-a77e-33ac5e69cf72","Type":"ContainerStarted","Data":"f7d1d69caf6bddf8497fb0282eb97513a7732016e5e48b4976187260438509d6"} Jul 24 17:43:34.784022 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.783965 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg" event={"ID":"72416b1c-f944-4c29-b8fe-5a3b053825f7","Type":"ContainerStarted","Data":"78f931a901e21b27a980c4b9b6b82268ec829175ceda9d803dc8f0bf6ddf24b3"} Jul 24 17:43:34.790689 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.790630 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" event={"ID":"c64451b6-8365-4a7a-8512-baaee772473d","Type":"ContainerStarted","Data":"8526b4426775ab1aac67a0a6a36ee3a54897c3dae595574eed6a9eed220ad94f"} Jul 24 17:43:34.796379 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.796318 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" event={"ID":"aebd90e8-411f-4c87-bed7-eb3a1ca82458","Type":"ContainerStarted","Data":"759707085ca285fe2a80708e99e22de293f89104382a00162d1b282ab5e0b446"} Jul 24 17:43:34.802887 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.802811 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp" event={"ID":"6690c968-3d7f-4ee5-9c7f-9d83510f7727","Type":"ContainerStarted","Data":"910379ae437049666ff436f7a0b0f68a886768a79932972ac3f9ff7d56d17d96"} Jul 24 17:43:34.806125 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:34.806058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" event={"ID":"a1c0f1b0-af8c-4b28-a678-386655111019","Type":"ContainerStarted","Data":"55b83aaed970287de8c0dd502bbb8d7d0aac2b8561df5ea228d8453dc11d4c1e"} Jul 24 17:43:35.719383 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:35.719134 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:35.719648 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:35.719588 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:35.719744 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:35.719671 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:35.719744 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:35.719709 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:35.719881 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:35.719745 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:35.719881 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:35.719809 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:35.720053 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.719993 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:43:35.720053 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720014 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f8b44f6-hkqlq: secret "image-registry-tls" not found Jul 24 17:43:35.720148 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720082 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls podName:fa94510b-f873-4c29-8b7b-668248d6e4cb nodeName:}" failed. No retries permitted until 2026-07-24 17:43:39.720064129 +0000 UTC m=+44.759858162 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls") pod "image-registry-658f8b44f6-hkqlq" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb") : secret "image-registry-tls" not found Jul 24 17:43:35.720616 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720569 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:43:35.720707 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720637 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:39.720619954 +0000 UTC m=+44.760413973 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : secret "router-metrics-certs-default" not found Jul 24 17:43:35.720771 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720709 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:43:35.720771 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720745 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls podName:96e88041-fc3f-4695-b1fc-6c3bbe62b3a9 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:39.720733133 +0000 UTC m=+44.760527167 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls") pod "cluster-samples-operator-6748489656-9xc5f" (UID: "96e88041-fc3f-4695-b1fc-6c3bbe62b3a9") : secret "samples-operator-tls" not found Jul 24 17:43:35.720917 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720835 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:39.720822042 +0000 UTC m=+44.760616063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : configmap references non-existent config key: service-ca.crt Jul 24 17:43:35.720917 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720894 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:35.721027 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720927 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls podName:02e6ee41-b12e-42ee-9589-f0435e5ae956 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:39.720916916 +0000 UTC m=+44.760710935 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-cvpqk" (UID: "02e6ee41-b12e-42ee-9589-f0435e5ae956") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:35.721027 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.720982 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:43:35.721027 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.721012 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls podName:d642fa46-5c47-4435-955b-1f73afb4fb81 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:39.721001775 +0000 UTC m=+44.760795798 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls") pod "dns-default-cbm78" (UID: "d642fa46-5c47-4435-955b-1f73afb4fb81") : secret "dns-default-metrics-tls" not found Jul 24 17:43:35.821155 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:35.820537 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:35.821155 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.820719 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 17:43:35.821155 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:35.820795 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert podName:edfb93c0-838e-40ee-8fda-324733002b82 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:39.820760299 +0000 UTC m=+44.860554322 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-26s9g" (UID: "edfb93c0-838e-40ee-8fda-324733002b82") : secret "networking-console-plugin-cert" not found Jul 24 17:43:35.843632 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:35.842427 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerStarted","Data":"1951bf99a0664d9c3780f9cca963ebd522252f449e23ffeb51a834319bb71dfd"} Jul 24 17:43:36.870128 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:36.869354 2559 generic.go:356] "Generic (PLEG): container finished" podID="0cab09e2-baba-4a01-a58f-53c91600107f" containerID="1951bf99a0664d9c3780f9cca963ebd522252f449e23ffeb51a834319bb71dfd" exitCode=0 Jul 24 17:43:36.870128 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:36.869450 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerDied","Data":"1951bf99a0664d9c3780f9cca963ebd522252f449e23ffeb51a834319bb71dfd"} Jul 24 17:43:39.764276 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:39.764240 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:39.764315 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:39.764357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:39.764395 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:39.764438 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764442 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764494 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:47.764468985 +0000 UTC m=+52.804263025 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : configmap references non-existent config key: service-ca.crt Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764519 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls podName:96e88041-fc3f-4695-b1fc-6c3bbe62b3a9 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:47.76451205 +0000 UTC m=+52.804306068 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls") pod "cluster-samples-operator-6748489656-9xc5f" (UID: "96e88041-fc3f-4695-b1fc-6c3bbe62b3a9") : secret "samples-operator-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764591 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764601 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764617 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f8b44f6-hkqlq: secret "image-registry-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764632 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764659 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls podName:fa94510b-f873-4c29-8b7b-668248d6e4cb nodeName:}" failed. No retries permitted until 2026-07-24 17:43:47.764644095 +0000 UTC m=+52.804438137 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls") pod "image-registry-658f8b44f6-hkqlq" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb") : secret "image-registry-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764679 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls podName:d642fa46-5c47-4435-955b-1f73afb4fb81 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:47.764669472 +0000 UTC m=+52.804463497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls") pod "dns-default-cbm78" (UID: "d642fa46-5c47-4435-955b-1f73afb4fb81") : secret "dns-default-metrics-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.764710 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls podName:02e6ee41-b12e-42ee-9589-f0435e5ae956 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:47.76470223 +0000 UTC m=+52.804496254 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-cvpqk" (UID: "02e6ee41-b12e-42ee-9589-f0435e5ae956") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:39.764775 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:39.764730 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:39.765445 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.765290 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:43:39.765445 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.765442 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:47.765418026 +0000 UTC m=+52.805212067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : secret "router-metrics-certs-default" not found Jul 24 17:43:39.866206 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:39.866170 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:39.866359 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.866333 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 17:43:39.866450 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:39.866428 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert podName:edfb93c0-838e-40ee-8fda-324733002b82 nodeName:}" failed. No retries permitted until 2026-07-24 17:43:47.866406383 +0000 UTC m=+52.906200408 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-26s9g" (UID: "edfb93c0-838e-40ee-8fda-324733002b82") : secret "networking-console-plugin-cert" not found Jul 24 17:43:47.840360 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.840327 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:43:47.840747 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.840393 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:47.840747 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.840427 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:43:47.840747 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.840463 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:43:47.840747 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.840511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:43:47.840747 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.840561 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:43:47.840747 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.840687 2559 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 24 17:43:47.840747 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.840738 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.840720434 +0000 UTC m=+68.880514458 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : secret "router-metrics-certs-default" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.840823 2559 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.840857 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls podName:96e88041-fc3f-4695-b1fc-6c3bbe62b3a9 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.840846611 +0000 UTC m=+68.880640644 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls") pod "cluster-samples-operator-6748489656-9xc5f" (UID: "96e88041-fc3f-4695-b1fc-6c3bbe62b3a9") : secret "samples-operator-tls" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.840920 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle podName:ed3b05b8-f53f-4148-8b50-dac538f09ad6 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.840909424 +0000 UTC m=+68.880703443 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle") pod "router-default-758d7d5dd5-wq9dp" (UID: "ed3b05b8-f53f-4148-8b50-dac538f09ad6") : configmap references non-existent config key: service-ca.crt Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.840987 2559 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.840987 2559 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.841018 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls podName:02e6ee41-b12e-42ee-9589-f0435e5ae956 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.841007532 +0000 UTC m=+68.880801552 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-cvpqk" (UID: "02e6ee41-b12e-42ee-9589-f0435e5ae956") : secret "cluster-monitoring-operator-tls" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.841033 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls podName:d642fa46-5c47-4435-955b-1f73afb4fb81 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.841024728 +0000 UTC m=+68.880818750 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls") pod "dns-default-cbm78" (UID: "d642fa46-5c47-4435-955b-1f73afb4fb81") : secret "dns-default-metrics-tls" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.841076 2559 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.841087 2559 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-658f8b44f6-hkqlq: secret "image-registry-tls" not found Jul 24 17:43:47.841159 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.841124 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls podName:fa94510b-f873-4c29-8b7b-668248d6e4cb nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.841112688 +0000 UTC m=+68.880906709 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls") pod "image-registry-658f8b44f6-hkqlq" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb") : secret "image-registry-tls" not found Jul 24 17:43:47.918557 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.917476 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-96j5z" event={"ID":"0cab09e2-baba-4a01-a58f-53c91600107f","Type":"ContainerStarted","Data":"fb98d64d018848ac41a424c77d4aad74a6069b42d61395fc2bdec3202e52038b"} Jul 24 17:43:47.923055 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.922837 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" event={"ID":"6f5e6ac9-6056-4018-a77e-33ac5e69cf72","Type":"ContainerStarted","Data":"c5a96f799c5a7902342023e5368aee0832df1c3268731297cd14c3b7bc10ca51"} Jul 24 17:43:47.924925 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.924846 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg" event={"ID":"72416b1c-f944-4c29-b8fe-5a3b053825f7","Type":"ContainerStarted","Data":"b10f508b15839647dfbc2b595e99ecb716703b440ccf63683aedb2ad3f573d9a"} Jul 24 17:43:47.928468 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.928442 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/0.log" Jul 24 17:43:47.928586 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.928484 2559 generic.go:356] "Generic (PLEG): container finished" podID="c64451b6-8365-4a7a-8512-baaee772473d" containerID="99dc3b3d7103a83542768a100e14856e658fc037531d376728d06bb422933f7f" exitCode=255 Jul 24 17:43:47.928586 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.928549 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" event={"ID":"c64451b6-8365-4a7a-8512-baaee772473d","Type":"ContainerDied","Data":"99dc3b3d7103a83542768a100e14856e658fc037531d376728d06bb422933f7f"} Jul 24 17:43:47.929169 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.928843 2559 scope.go:117] "RemoveContainer" containerID="99dc3b3d7103a83542768a100e14856e658fc037531d376728d06bb422933f7f" Jul 24 17:43:47.939953 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.938943 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" event={"ID":"aebd90e8-411f-4c87-bed7-eb3a1ca82458","Type":"ContainerStarted","Data":"6f098caaefa9fbaff882ca37621452bfe73c3cd9050b25f84f320a865a821239"} Jul 24 17:43:47.941530 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.941507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:43:47.944180 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.944161 2559 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 24 17:43:47.944334 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:47.944324 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert podName:edfb93c0-838e-40ee-8fda-324733002b82 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:03.944303347 +0000 UTC m=+68.984097368 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert") pod "networking-console-plugin-74c9f884b7-26s9g" (UID: "edfb93c0-838e-40ee-8fda-324733002b82") : secret "networking-console-plugin-cert" not found Jul 24 17:43:47.950902 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.948451 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-96j5z" podStartSLOduration=21.849089999 podStartE2EDuration="52.948432455s" totalImagesPullingTime="31.099342456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:55 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:47.945698152 +0000 UTC m=+52.985492194" watchObservedRunningTime="2026-07-24 17:43:47.948432455 +0000 UTC m=+52.988226499" Jul 24 17:43:47.950902 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.949365 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp" event={"ID":"6690c968-3d7f-4ee5-9c7f-9d83510f7727","Type":"ContainerStarted","Data":"fff5d8af71bb6043ae779fc3a3d4161297176ea44d03ca6f4b2e91666c7813b7"} Jul 24 17:43:47.954490 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.954447 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" event={"ID":"88676e21-31ea-4fdf-963d-04c2e0fb7c75","Type":"ContainerStarted","Data":"6ab88f30fa337b2350365369ed6fd11f7384ea775e51ec696beabf1ae4481b24"} Jul 24 17:43:47.957116 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.956179 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" event={"ID":"a671467a-5ae1-4531-86ec-f36ea72cd2a0","Type":"ContainerStarted","Data":"cdfdef71e60b4a1cfb5f604f49583e81d25ee61f6877395d3a8b956e502e7290"} Jul 24 17:43:47.957116 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.956924 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:47.961769 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.961654 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" Jul 24 17:43:47.962457 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.962377 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-k2526" event={"ID":"b9c8447a-8f08-4d76-a7d3-103f60b7e495","Type":"ContainerStarted","Data":"5c137d650aca39918359d5a1c2d23527676a160ee28a1395c4af43a273908258"} Jul 24 17:43:47.967253 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.967232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hnvlj" event={"ID":"f5e968a2-537a-4759-bc98-e669d89fe627","Type":"ContainerStarted","Data":"3bd145faab276d9fd47a67df00ade536c177aa87b146285e487f324538a8e630"} Jul 24 17:43:47.993180 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.991681 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-wqnfg" podStartSLOduration=25.93094118 podStartE2EDuration="35.991661459s" totalImagesPullingTime="10.060720279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:47.966495957 +0000 UTC m=+53.006289999" watchObservedRunningTime="2026-07-24 17:43:47.991661459 +0000 UTC m=+53.031455502" Jul 24 17:43:47.993180 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:47.993019 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" podStartSLOduration=22.363399038 podStartE2EDuration="35.993005698s" totalImagesPullingTime="13.62960666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:47.991296718 +0000 UTC m=+53.031090761" watchObservedRunningTime="2026-07-24 17:43:47.993005698 +0000 UTC m=+53.032799741" Jul 24 17:43:48.061361 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.061245 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hnvlj" podStartSLOduration=3.431333195 podStartE2EDuration="17.061223671s" totalImagesPullingTime="13.629890476s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:48.060542456 +0000 UTC m=+53.100336492" watchObservedRunningTime="2026-07-24 17:43:48.061223671 +0000 UTC m=+53.101017705" Jul 24 17:43:48.062306 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.062269 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" podStartSLOduration=22.97121619 podStartE2EDuration="36.062251006s" totalImagesPullingTime="13.091034816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:48.038960641 +0000 UTC m=+53.078754682" watchObservedRunningTime="2026-07-24 17:43:48.062251006 +0000 UTC m=+53.102045042" Jul 24 17:43:48.086569 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.085994 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5885fdb759-k2526" podStartSLOduration=22.993406287 podStartE2EDuration="36.085971317s" totalImagesPullingTime="13.09256503s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:48.085296561 +0000 UTC m=+53.125090602" watchObservedRunningTime="2026-07-24 17:43:48.085971317 +0000 UTC m=+53.125765360" Jul 24 17:43:48.132528 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.132474 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-84ff5444c6-h5z2g" podStartSLOduration=13.418028624 podStartE2EDuration="27.132455125s" totalImagesPullingTime="13.714426501s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:48.105037228 +0000 UTC m=+53.144831262" watchObservedRunningTime="2026-07-24 17:43:48.132455125 +0000 UTC m=+53.172249178" Jul 24 17:43:48.132707 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.132677 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7d88cb55d8-zszlp" podStartSLOduration=22.418838063 podStartE2EDuration="36.132664415s" totalImagesPullingTime="13.713826352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:48.131023681 +0000 UTC m=+53.170817723" watchObservedRunningTime="2026-07-24 17:43:48.132664415 +0000 UTC m=+53.172458448" Jul 24 17:43:48.974662 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.974232 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" event={"ID":"a1c0f1b0-af8c-4b28-a678-386655111019","Type":"ContainerStarted","Data":"562b81e2147b926a481a4d248ff5013df1c3f1763c0aa783a00e29f91d0ebfe3"} Jul 24 17:43:48.977710 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.977273 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" event={"ID":"88676e21-31ea-4fdf-963d-04c2e0fb7c75","Type":"ContainerStarted","Data":"c0810f69a676d54c6f8d8a4d32797d3dee002049259c2e7bdcd0fee84fe4d7e3"} Jul 24 17:43:48.977710 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.977319 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" event={"ID":"88676e21-31ea-4fdf-963d-04c2e0fb7c75","Type":"ContainerStarted","Data":"814380ba6964aa53c3631028f893ac03ce051960920b08504d1628df0a72e51e"} Jul 24 17:43:48.979145 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.979120 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:43:48.979729 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.979707 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/0.log" Jul 24 17:43:48.979864 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.979752 2559 generic.go:356] "Generic (PLEG): container finished" podID="c64451b6-8365-4a7a-8512-baaee772473d" containerID="ecb6b71d64c10879f5adb16ee1f9f390ea0bc51fce16a8f5bb4a1d821ba4ab10" exitCode=255 Jul 24 17:43:48.979924 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.979860 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" event={"ID":"c64451b6-8365-4a7a-8512-baaee772473d","Type":"ContainerDied","Data":"ecb6b71d64c10879f5adb16ee1f9f390ea0bc51fce16a8f5bb4a1d821ba4ab10"} Jul 24 17:43:48.979924 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.979914 2559 scope.go:117] "RemoveContainer" containerID="99dc3b3d7103a83542768a100e14856e658fc037531d376728d06bb422933f7f" Jul 24 17:43:48.980198 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.980182 2559 scope.go:117] "RemoveContainer" containerID="ecb6b71d64c10879f5adb16ee1f9f390ea0bc51fce16a8f5bb4a1d821ba4ab10" Jul 24 17:43:48.980375 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:48.980346 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-dz2pl_openshift-console-operator(c64451b6-8365-4a7a-8512-baaee772473d)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" podUID="c64451b6-8365-4a7a-8512-baaee772473d" Jul 24 17:43:48.997045 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:48.996991 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5969db5bc4-2qb6b" podStartSLOduration=14.272045811 podStartE2EDuration="27.996972575s" totalImagesPullingTime="13.724926764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:48.996813718 +0000 UTC m=+54.036607760" watchObservedRunningTime="2026-07-24 17:43:48.996972575 +0000 UTC m=+54.036766617" Jul 24 17:43:49.002255 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.002230 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-24c26"] Jul 24 17:43:49.023039 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.022980 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" podStartSLOduration=14.298970768 podStartE2EDuration="28.022960559s" totalImagesPullingTime="13.723989791s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:43:49.021036542 +0000 UTC m=+54.060830584" watchObservedRunningTime="2026-07-24 17:43:49.022960559 +0000 UTC m=+54.062754602" Jul 24 17:43:49.035059 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.035033 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24c26"] Jul 24 17:43:49.035268 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.035125 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.038006 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.037984 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 24 17:43:49.038006 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.038001 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 24 17:43:49.038151 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.037984 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cg7vk\"" Jul 24 17:43:49.157063 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.157031 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h27h\" (UniqueName: \"kubernetes.io/projected/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-api-access-9h27h\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.157063 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.157076 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.157315 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.157114 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6bce77f-3e2f-40af-8109-b52446e53c4a-data-volume\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.157315 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.157180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.157315 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.157249 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6bce77f-3e2f-40af-8109-b52446e53c4a-crio-socket\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.258286 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.258200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6bce77f-3e2f-40af-8109-b52446e53c4a-data-volume\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.258286 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.258238 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.258286 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.258270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6bce77f-3e2f-40af-8109-b52446e53c4a-crio-socket\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.258565 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.258370 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:43:49.258565 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.258428 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h27h\" (UniqueName: \"kubernetes.io/projected/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-api-access-9h27h\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.258565 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.258427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d6bce77f-3e2f-40af-8109-b52446e53c4a-crio-socket\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.258565 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.258448 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:49.758430773 +0000 UTC m=+54.798224791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : secret "insights-runtime-extractor-tls" not found Jul 24 17:43:49.258565 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.258499 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.258759 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.258589 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:43:49.258759 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.258597 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d6bce77f-3e2f-40af-8109-b52446e53c4a-data-volume\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.258759 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.258632 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:49.758620302 +0000 UTC m=+54.798414326 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : configmap "kube-rbac-proxy" not found Jul 24 17:43:49.271485 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.271459 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h27h\" (UniqueName: \"kubernetes.io/projected/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-api-access-9h27h\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.762544 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.762507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.762759 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.762562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:49.762759 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.762628 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:43:49.762759 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.762700 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:50.762682245 +0000 UTC m=+55.802476264 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : configmap "kube-rbac-proxy" not found Jul 24 17:43:49.762759 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.762732 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:43:49.762931 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.762821 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:50.762779634 +0000 UTC m=+55.802573653 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : secret "insights-runtime-extractor-tls" not found Jul 24 17:43:49.985297 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.985264 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:43:49.985759 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:49.985723 2559 scope.go:117] "RemoveContainer" containerID="ecb6b71d64c10879f5adb16ee1f9f390ea0bc51fce16a8f5bb4a1d821ba4ab10" Jul 24 17:43:49.985960 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:49.985931 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-dz2pl_openshift-console-operator(c64451b6-8365-4a7a-8512-baaee772473d)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" podUID="c64451b6-8365-4a7a-8512-baaee772473d" Jul 24 17:43:50.773296 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:50.773262 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:50.773488 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:50.773318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:50.773488 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:50.773380 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:43:50.773488 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:50.773452 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:52.773435067 +0000 UTC m=+57.813229087 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : configmap "kube-rbac-proxy" not found Jul 24 17:43:50.773488 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:50.773476 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:43:50.773631 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:50.773544 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:52.773527222 +0000 UTC m=+57.813321269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : secret "insights-runtime-extractor-tls" not found Jul 24 17:43:50.860182 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:50.860153 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gnjpf_3076bbf1-a934-4c5d-a679-a3aa1c824daf/dns-node-resolver/0.log" Jul 24 17:43:52.060223 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:52.060196 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fsf7z_b00377e3-0fe2-4fe7-89cb-c892e233959f/node-ca/0.log" Jul 24 17:43:52.292094 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:52.292064 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:52.292256 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:52.292114 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:43:52.292439 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:52.292428 2559 scope.go:117] "RemoveContainer" containerID="ecb6b71d64c10879f5adb16ee1f9f390ea0bc51fce16a8f5bb4a1d821ba4ab10" Jul 24 17:43:52.292590 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:52.292572 2559 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-dz2pl_openshift-console-operator(c64451b6-8365-4a7a-8512-baaee772473d)\"" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" podUID="c64451b6-8365-4a7a-8512-baaee772473d" Jul 24 17:43:52.791275 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:52.791223 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:52.791456 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:52.791300 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:52.791456 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:52.791373 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:43:52.791456 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:52.791412 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:43:52.791456 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:52.791456 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:56.791439199 +0000 UTC m=+61.831233218 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : configmap "kube-rbac-proxy" not found Jul 24 17:43:52.791593 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:52.791471 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:43:56.791465231 +0000 UTC m=+61.831259250 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : secret "insights-runtime-extractor-tls" not found Jul 24 17:43:52.860286 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:52.860250 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hnvlj_f5e968a2-537a-4759-bc98-e669d89fe627/serve-healthcheck-canary/0.log" Jul 24 17:43:56.703132 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:56.703100 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-bk2sq" Jul 24 17:43:56.828622 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:56.828586 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:56.828835 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:56.828655 2559 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 24 17:43:56.828835 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:56.828706 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:44:04.828691825 +0000 UTC m=+69.868485843 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : configmap "kube-rbac-proxy" not found Jul 24 17:43:56.828958 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:43:56.828840 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:43:56.828958 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:56.828942 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:43:56.829032 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:43:56.829004 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:44:04.828987745 +0000 UTC m=+69.868781765 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : secret "insights-runtime-extractor-tls" not found Jul 24 17:44:01.166999 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.166957 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:44:01.170067 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.170041 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 24 17:44:01.177642 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:44:01.177620 2559 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 24 17:44:01.177750 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:44:01.177693 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs podName:1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a nodeName:}" failed. No retries permitted until 2026-07-24 17:45:05.177670858 +0000 UTC m=+130.217464878 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs") pod "network-metrics-daemon-5pvds" (UID: "1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a") : secret "metrics-daemon-secret" not found Jul 24 17:44:01.268280 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.268246 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:44:01.270997 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.270972 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 24 17:44:01.280756 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.280730 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b88c2b38-14f5-4ef4-8e28-76663ce9e34f-original-pull-secret\") pod \"global-pull-secret-syncer-zh5f7\" (UID: \"b88c2b38-14f5-4ef4-8e28-76663ce9e34f\") " pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:44:01.368982 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.368944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:44:01.371563 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.371532 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzcln\" (UniqueName: \"kubernetes.io/projected/9aeff889-3a93-4ecb-a76d-925dfa85e4fa-kube-api-access-dzcln\") pod \"network-check-target-26mkw\" (UID: \"9aeff889-3a93-4ecb-a76d-925dfa85e4fa\") " pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:44:01.520251 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.520162 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-9txdv\"" Jul 24 17:44:01.528408 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.528378 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:44:01.547762 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.547736 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zh5f7" Jul 24 17:44:01.663502 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.663472 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-26mkw"] Jul 24 17:44:01.666931 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:44:01.666900 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aeff889_3a93_4ecb_a76d_925dfa85e4fa.slice/crio-a2ac201487bb2b69034b793d6590aa2805f1e7e87114a92d222936eacb5f00c4 WatchSource:0}: Error finding container a2ac201487bb2b69034b793d6590aa2805f1e7e87114a92d222936eacb5f00c4: Status 404 returned error can't find the container with id a2ac201487bb2b69034b793d6590aa2805f1e7e87114a92d222936eacb5f00c4 Jul 24 17:44:01.681646 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:01.681622 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zh5f7"] Jul 24 17:44:01.684824 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:44:01.684768 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb88c2b38_14f5_4ef4_8e28_76663ce9e34f.slice/crio-1998648818c8f8ecf3e966b94f9c15e0e2d5053548776e786fcf1f1219986518 WatchSource:0}: Error finding container 1998648818c8f8ecf3e966b94f9c15e0e2d5053548776e786fcf1f1219986518: Status 404 returned error can't find the container with id 1998648818c8f8ecf3e966b94f9c15e0e2d5053548776e786fcf1f1219986518 Jul 24 17:44:02.028000 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:02.027961 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26mkw" event={"ID":"9aeff889-3a93-4ecb-a76d-925dfa85e4fa","Type":"ContainerStarted","Data":"e6fa026cac0f0624ce35ec7158f6b18b37eb324ee04b5b17e2edc71b5acc3157"} Jul 24 17:44:02.028181 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:02.028013 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-26mkw" event={"ID":"9aeff889-3a93-4ecb-a76d-925dfa85e4fa","Type":"ContainerStarted","Data":"a2ac201487bb2b69034b793d6590aa2805f1e7e87114a92d222936eacb5f00c4"} Jul 24 17:44:02.028181 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:02.028062 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:44:02.029224 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:02.029204 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zh5f7" event={"ID":"b88c2b38-14f5-4ef4-8e28-76663ce9e34f","Type":"ContainerStarted","Data":"1998648818c8f8ecf3e966b94f9c15e0e2d5053548776e786fcf1f1219986518"} Jul 24 17:44:02.045504 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:02.045444 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-26mkw" podStartSLOduration=66.045419551 podStartE2EDuration="1m6.045419551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:02.044473501 +0000 UTC m=+67.084267542" watchObservedRunningTime="2026-07-24 17:44:02.045419551 +0000 UTC m=+67.085213591" Jul 24 17:44:03.892318 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.892270 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:03.892832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.892323 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:44:03.892832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.892539 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:44:03.892832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.892618 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:44:03.892832 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.892678 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:03.893079 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.892908 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:44:03.893079 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.893062 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed3b05b8-f53f-4148-8b50-dac538f09ad6-service-ca-bundle\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:03.895759 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.895726 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d642fa46-5c47-4435-955b-1f73afb4fb81-metrics-tls\") pod \"dns-default-cbm78\" (UID: \"d642fa46-5c47-4435-955b-1f73afb4fb81\") " pod="openshift-dns/dns-default-cbm78" Jul 24 17:44:03.896047 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.896024 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ed3b05b8-f53f-4148-8b50-dac538f09ad6-metrics-certs\") pod \"router-default-758d7d5dd5-wq9dp\" (UID: \"ed3b05b8-f53f-4148-8b50-dac538f09ad6\") " pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:03.896138 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.896055 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/96e88041-fc3f-4695-b1fc-6c3bbe62b3a9-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-9xc5f\" (UID: \"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:44:03.896198 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.896162 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/02e6ee41-b12e-42ee-9589-f0435e5ae956-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-cvpqk\" (UID: \"02e6ee41-b12e-42ee-9589-f0435e5ae956\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:44:03.897149 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.897130 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"image-registry-658f8b44f6-hkqlq\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:44:03.993743 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.993702 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:44:03.996746 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:03.996716 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/edfb93c0-838e-40ee-8fda-324733002b82-networking-console-plugin-cert\") pod \"networking-console-plugin-74c9f884b7-26s9g\" (UID: \"edfb93c0-838e-40ee-8fda-324733002b82\") " pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:44:04.004521 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.004499 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vh2n6\"" Jul 24 17:44:04.011657 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.011626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:44:04.068644 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.068614 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-hddhl\"" Jul 24 17:44:04.076954 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.076850 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" Jul 24 17:44:04.086944 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.086913 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5kpm4\"" Jul 24 17:44:04.094416 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.094387 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-cbm78" Jul 24 17:44:04.110762 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.110490 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-sxl67\"" Jul 24 17:44:04.117376 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.117309 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dwmxq\"" Jul 24 17:44:04.118294 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.118270 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" Jul 24 17:44:04.125477 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.125455 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:04.158761 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.158634 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-982db\"" Jul 24 17:44:04.167352 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.166909 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" Jul 24 17:44:04.216616 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.214895 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-658f8b44f6-hkqlq"] Jul 24 17:44:04.295368 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.295277 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk"] Jul 24 17:44:04.318881 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:44:04.318825 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02e6ee41_b12e_42ee_9589_f0435e5ae956.slice/crio-8afe354ee246249d6f83d395f25ddeba57ca65ac0e2dd6c0eb2179ee7c27f142 WatchSource:0}: Error finding container 8afe354ee246249d6f83d395f25ddeba57ca65ac0e2dd6c0eb2179ee7c27f142: Status 404 returned error can't find the container with id 8afe354ee246249d6f83d395f25ddeba57ca65ac0e2dd6c0eb2179ee7c27f142 Jul 24 17:44:04.374386 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.374351 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-cbm78"] Jul 24 17:44:04.401308 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.401188 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f"] Jul 24 17:44:04.426471 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.425852 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-758d7d5dd5-wq9dp"] Jul 24 17:44:04.432023 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:44:04.431989 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3b05b8_f53f_4148_8b50_dac538f09ad6.slice/crio-9515c7b94c6d55566cd475844c14911abc9eb2f5cb91a51b39f1e60df730afd4 WatchSource:0}: Error finding container 9515c7b94c6d55566cd475844c14911abc9eb2f5cb91a51b39f1e60df730afd4: Status 404 returned error can't find the container with id 9515c7b94c6d55566cd475844c14911abc9eb2f5cb91a51b39f1e60df730afd4 Jul 24 17:44:04.444142 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.444117 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-74c9f884b7-26s9g"] Jul 24 17:44:04.447854 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:44:04.447825 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedfb93c0_838e_40ee_8fda_324733002b82.slice/crio-5aba975141301858dd9fa5000d8dc495433285df20ee274be4e19e1d1536a918 WatchSource:0}: Error finding container 5aba975141301858dd9fa5000d8dc495433285df20ee274be4e19e1d1536a918: Status 404 returned error can't find the container with id 5aba975141301858dd9fa5000d8dc495433285df20ee274be4e19e1d1536a918 Jul 24 17:44:04.912077 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.912039 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:44:04.912529 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.912092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:44:04.912529 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:44:04.912191 2559 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 24 17:44:04.912529 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:44:04.912254 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls podName:d6bce77f-3e2f-40af-8109-b52446e53c4a nodeName:}" failed. No retries permitted until 2026-07-24 17:44:20.912235551 +0000 UTC m=+85.952029573 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-24c26" (UID: "d6bce77f-3e2f-40af-8109-b52446e53c4a") : secret "insights-runtime-extractor-tls" not found Jul 24 17:44:04.912692 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:04.912673 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d6bce77f-3e2f-40af-8109-b52446e53c4a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:44:05.043406 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.043370 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbm78" event={"ID":"d642fa46-5c47-4435-955b-1f73afb4fb81","Type":"ContainerStarted","Data":"faf29b3731e767827fc28b776806d6b4bfe9c485a67ef59bb58c482e917c4065"} Jul 24 17:44:05.044749 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.044690 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" event={"ID":"02e6ee41-b12e-42ee-9589-f0435e5ae956","Type":"ContainerStarted","Data":"8afe354ee246249d6f83d395f25ddeba57ca65ac0e2dd6c0eb2179ee7c27f142"} Jul 24 17:44:05.046546 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.046521 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" event={"ID":"fa94510b-f873-4c29-8b7b-668248d6e4cb","Type":"ContainerStarted","Data":"265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886"} Jul 24 17:44:05.046671 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.046565 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" event={"ID":"fa94510b-f873-4c29-8b7b-668248d6e4cb","Type":"ContainerStarted","Data":"c4ced713aa5dc4f41675214b9a8a9a69e3616f9a9a75d917f65aa0e6572529a2"} Jul 24 17:44:05.046671 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.046617 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:44:05.047857 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.047830 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" event={"ID":"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9","Type":"ContainerStarted","Data":"546602aaa1ed3ae2bc3a1c4b29cc039ea273c503a9705dc1cdeba6cf89202867"} Jul 24 17:44:05.049613 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.049585 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" event={"ID":"ed3b05b8-f53f-4148-8b50-dac538f09ad6","Type":"ContainerStarted","Data":"059d6716d9a67257795937a5ee02aaaee1ea27eea9ab0b524d050f9eb952029b"} Jul 24 17:44:05.049731 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.049632 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" event={"ID":"ed3b05b8-f53f-4148-8b50-dac538f09ad6","Type":"ContainerStarted","Data":"9515c7b94c6d55566cd475844c14911abc9eb2f5cb91a51b39f1e60df730afd4"} Jul 24 17:44:05.051024 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.051003 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" event={"ID":"edfb93c0-838e-40ee-8fda-324733002b82","Type":"ContainerStarted","Data":"5aba975141301858dd9fa5000d8dc495433285df20ee274be4e19e1d1536a918"} Jul 24 17:44:05.069656 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.069600 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" podStartSLOduration=69.069584643 podStartE2EDuration="1m9.069584643s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:05.067926136 +0000 UTC m=+70.107720176" watchObservedRunningTime="2026-07-24 17:44:05.069584643 +0000 UTC m=+70.109378684" Jul 24 17:44:05.087225 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.087177 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" podStartSLOduration=53.087160893 podStartE2EDuration="53.087160893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:05.086350428 +0000 UTC m=+70.126144469" watchObservedRunningTime="2026-07-24 17:44:05.087160893 +0000 UTC m=+70.126954925" Jul 24 17:44:05.125950 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.125919 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:05.128905 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.128880 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:05.490764 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:05.490735 2559 scope.go:117] "RemoveContainer" containerID="ecb6b71d64c10879f5adb16ee1f9f390ea0bc51fce16a8f5bb4a1d821ba4ab10" Jul 24 17:44:06.055382 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:06.055342 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:06.056479 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:06.056458 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-758d7d5dd5-wq9dp" Jul 24 17:44:07.065097 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:07.063838 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:44:07.065097 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:07.063954 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" event={"ID":"c64451b6-8365-4a7a-8512-baaee772473d","Type":"ContainerStarted","Data":"b1f29220b827024f7e5b9f1f6b81d08be6227af907fa6a0e0b9bd3dd05aa9c0f"} Jul 24 17:44:07.065097 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:07.065043 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:44:07.068813 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:07.068736 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zh5f7" event={"ID":"b88c2b38-14f5-4ef4-8e28-76663ce9e34f","Type":"ContainerStarted","Data":"969ffd55479d2852511025f4ea464541fe2795bbd0d40b12390ab80b421e8b23"} Jul 24 17:44:07.072105 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:07.072082 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" Jul 24 17:44:07.083023 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:07.082970 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-dz2pl" podStartSLOduration=41.451160294 podStartE2EDuration="55.082951471s" totalImagesPullingTime="13.631791177s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:07.082195845 +0000 UTC m=+72.121989887" watchObservedRunningTime="2026-07-24 17:44:07.082951471 +0000 UTC m=+72.122745513" Jul 24 17:44:07.096802 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:07.096692 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zh5f7" podStartSLOduration=66.631096369 podStartE2EDuration="1m11.096673596s" totalImagesPullingTime="4.465577227s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:42:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:07.096477278 +0000 UTC m=+72.136271320" watchObservedRunningTime="2026-07-24 17:44:07.096673596 +0000 UTC m=+72.136467637" Jul 24 17:44:08.074195 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:08.074157 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" event={"ID":"edfb93c0-838e-40ee-8fda-324733002b82","Type":"ContainerStarted","Data":"ea475f1f6db7fa6704eaf7bb26188a7d2ae23e2a6ec748138fd5628656fe5277"} Jul 24 17:44:08.091422 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:08.091363 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-74c9f884b7-26s9g" podStartSLOduration=46.278862723 podStartE2EDuration="49.091338203s" totalImagesPullingTime="2.81247548s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:08.091012033 +0000 UTC m=+73.130806087" watchObservedRunningTime="2026-07-24 17:44:08.091338203 +0000 UTC m=+73.131132226" Jul 24 17:44:09.079894 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:09.079802 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" event={"ID":"02e6ee41-b12e-42ee-9589-f0435e5ae956","Type":"ContainerStarted","Data":"2d5afcc35a121f7364fa8a2f6fd5383507b97cc325de6096d61785d679678aee"} Jul 24 17:44:09.082104 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:09.082074 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" event={"ID":"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9","Type":"ContainerStarted","Data":"02816b69bb730ac08160e202920c7dff40747d05e3928a23e3dc75da8d2abd0b"} Jul 24 17:44:09.082234 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:09.082131 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" event={"ID":"96e88041-fc3f-4695-b1fc-6c3bbe62b3a9","Type":"ContainerStarted","Data":"f6f7c9e801d631f41b7a176748dfa596018fca71aca5bae14fc913a2552ccea6"} Jul 24 17:44:09.084108 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:09.084089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbm78" event={"ID":"d642fa46-5c47-4435-955b-1f73afb4fb81","Type":"ContainerStarted","Data":"50f25d0151fbb0b8272287b954d297df1ea0b9e83b95c43747f8148404d75ec3"} Jul 24 17:44:09.084217 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:09.084121 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-cbm78" event={"ID":"d642fa46-5c47-4435-955b-1f73afb4fb81","Type":"ContainerStarted","Data":"d756f537759bdd7a09a6c60f0778728749937211aa25ec92a2374ae9c0d45a43"} Jul 24 17:44:09.097978 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:09.097926 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-cvpqk" podStartSLOduration=52.577442288 podStartE2EDuration="57.097906522s" totalImagesPullingTime="4.520464234s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:09.096605544 +0000 UTC m=+74.136399586" watchObservedRunningTime="2026-07-24 17:44:09.097906522 +0000 UTC m=+74.137700541" Jul 24 17:44:09.115334 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:09.115282 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-cbm78" podStartSLOduration=33.662906102 podStartE2EDuration="38.11526085s" totalImagesPullingTime="4.452354748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:09.114689272 +0000 UTC m=+74.154483315" watchObservedRunningTime="2026-07-24 17:44:09.11526085 +0000 UTC m=+74.155054892" Jul 24 17:44:09.136083 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:09.136026 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-9xc5f" podStartSLOduration=52.784170516 podStartE2EDuration="57.136005331s" totalImagesPullingTime="4.351834815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:09.134731883 +0000 UTC m=+74.174525921" watchObservedRunningTime="2026-07-24 17:44:09.136005331 +0000 UTC m=+74.175799374" Jul 24 17:44:10.088227 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:10.088190 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-cbm78" Jul 24 17:44:20.093651 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:20.093548 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-cbm78" Jul 24 17:44:20.963225 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:20.963190 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:44:20.965604 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:20.965587 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d6bce77f-3e2f-40af-8109-b52446e53c4a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-24c26\" (UID: \"d6bce77f-3e2f-40af-8109-b52446e53c4a\") " pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:44:21.149212 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:21.149185 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cg7vk\"" Jul 24 17:44:21.157269 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:21.157244 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-24c26" Jul 24 17:44:21.318401 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:21.318369 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-24c26"] Jul 24 17:44:21.321119 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:44:21.321089 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6bce77f_3e2f_40af_8109_b52446e53c4a.slice/crio-ba4a1a2477d2bb82e98b89e8708bdef57394617ed93589fbdc9dbf5d05057874 WatchSource:0}: Error finding container ba4a1a2477d2bb82e98b89e8708bdef57394617ed93589fbdc9dbf5d05057874: Status 404 returned error can't find the container with id ba4a1a2477d2bb82e98b89e8708bdef57394617ed93589fbdc9dbf5d05057874 Jul 24 17:44:22.134411 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.134326 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24c26" event={"ID":"d6bce77f-3e2f-40af-8109-b52446e53c4a","Type":"ContainerStarted","Data":"ba4a1a2477d2bb82e98b89e8708bdef57394617ed93589fbdc9dbf5d05057874"} Jul 24 17:44:22.816916 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.816887 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-slxvw"] Jul 24 17:44:22.821410 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.821384 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.824216 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.824185 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 24 17:44:22.824493 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.824470 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mfdvq\"" Jul 24 17:44:22.824622 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.824472 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 24 17:44:22.825178 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.825154 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 24 17:44:22.825295 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.825164 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 24 17:44:22.878988 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.878958 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-sys\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.879167 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.879006 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-accelerators-collector-config\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.879167 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.879071 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-tls\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.879167 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.879130 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-textfile\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.879167 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.879158 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-wtmp\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.879377 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.879200 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-metrics-client-ca\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.879377 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.879260 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-root\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.879377 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.879285 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47bhj\" (UniqueName: \"kubernetes.io/projected/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-kube-api-access-47bhj\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.879377 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.879314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980140 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980108 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-textfile\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980140 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980142 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-wtmp\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980367 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980176 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-metrics-client-ca\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980367 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980224 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-root\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980367 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47bhj\" (UniqueName: \"kubernetes.io/projected/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-kube-api-access-47bhj\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980367 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980367 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980311 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-sys\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980367 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-accelerators-collector-config\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980640 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980372 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-wtmp\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980640 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980375 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-tls\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980640 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980449 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-root\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980640 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980473 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-sys\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980640 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:44:22.980553 2559 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 24 17:44:22.980640 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:44:22.980612 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-tls podName:20622eb3-3ce9-498a-b95a-fdc5c59b7e71 nodeName:}" failed. No retries permitted until 2026-07-24 17:44:23.480592551 +0000 UTC m=+88.520386575 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-tls") pod "node-exporter-slxvw" (UID: "20622eb3-3ce9-498a-b95a-fdc5c59b7e71") : secret "node-exporter-tls" not found Jul 24 17:44:22.980982 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980662 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-textfile\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.980982 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.980886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-metrics-client-ca\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.981089 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.981034 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-accelerators-collector-config\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.983086 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.983064 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:22.989029 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:22.989004 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47bhj\" (UniqueName: \"kubernetes.io/projected/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-kube-api-access-47bhj\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:23.485444 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:23.485399 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-tls\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:23.488074 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:23.488049 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/20622eb3-3ce9-498a-b95a-fdc5c59b7e71-node-exporter-tls\") pod \"node-exporter-slxvw\" (UID: \"20622eb3-3ce9-498a-b95a-fdc5c59b7e71\") " pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:23.733607 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:23.733571 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-slxvw" Jul 24 17:44:24.016655 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:24.016571 2559 patch_prober.go:28] interesting pod/image-registry-658f8b44f6-hkqlq container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 24 17:44:24.016655 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:24.016626 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" podUID="fa94510b-f873-4c29-8b7b-668248d6e4cb" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 17:44:26.060306 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:26.060276 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:44:26.302637 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:44:26.302605 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20622eb3_3ce9_498a_b95a_fdc5c59b7e71.slice/crio-f69067d39de8b6703141e0bee7ee1d43ca7e2a0cb99ba73c15d9ad5a99a51e49 WatchSource:0}: Error finding container f69067d39de8b6703141e0bee7ee1d43ca7e2a0cb99ba73c15d9ad5a99a51e49: Status 404 returned error can't find the container with id f69067d39de8b6703141e0bee7ee1d43ca7e2a0cb99ba73c15d9ad5a99a51e49 Jul 24 17:44:27.163619 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:27.163575 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slxvw" event={"ID":"20622eb3-3ce9-498a-b95a-fdc5c59b7e71","Type":"ContainerStarted","Data":"f69067d39de8b6703141e0bee7ee1d43ca7e2a0cb99ba73c15d9ad5a99a51e49"} Jul 24 17:44:27.165218 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:27.165189 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24c26" event={"ID":"d6bce77f-3e2f-40af-8109-b52446e53c4a","Type":"ContainerStarted","Data":"620fdbe8405bd552711a6d145abe0256ffa1f906243edfa502517765e07c0fb2"} Jul 24 17:44:28.171449 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:28.171399 2559 generic.go:356] "Generic (PLEG): container finished" podID="20622eb3-3ce9-498a-b95a-fdc5c59b7e71" containerID="c630df0fe61241b797b476e74f8771002d0b7f630eed4035344b59e82e72cd08" exitCode=0 Jul 24 17:44:28.171946 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:28.171461 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slxvw" event={"ID":"20622eb3-3ce9-498a-b95a-fdc5c59b7e71","Type":"ContainerDied","Data":"c630df0fe61241b797b476e74f8771002d0b7f630eed4035344b59e82e72cd08"} Jul 24 17:44:30.180723 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:30.180619 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24c26" event={"ID":"d6bce77f-3e2f-40af-8109-b52446e53c4a","Type":"ContainerStarted","Data":"8c5eaeda3b9fc83a9ae602546c05f254f57a5cddf414fc1a34ff081ee343458d"} Jul 24 17:44:30.182633 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:30.182607 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slxvw" event={"ID":"20622eb3-3ce9-498a-b95a-fdc5c59b7e71","Type":"ContainerStarted","Data":"902fb537cf38ddc40ba0af0a41216fd15d4230f4e3d93d07b48a60eb3efc3095"} Jul 24 17:44:30.182764 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:30.182652 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-slxvw" event={"ID":"20622eb3-3ce9-498a-b95a-fdc5c59b7e71","Type":"ContainerStarted","Data":"71ba1d21f7286e6b8127c920372fae71161a33b30039604e1df97626c5abf20a"} Jul 24 17:44:30.204297 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:30.204222 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-slxvw" podStartSLOduration=7.117108718 podStartE2EDuration="8.20420741s" totalImagesPullingTime="994.351277ms" totalInitContainerRuntime="92.747415ms" isStatefulPod=true podCreationTimestamp="2026-07-24 17:44:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:30.202351893 +0000 UTC m=+95.242145934" watchObservedRunningTime="2026-07-24 17:44:30.20420741 +0000 UTC m=+95.244001428" Jul 24 17:44:33.036949 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:33.036922 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-26mkw" Jul 24 17:44:34.202726 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:34.202692 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-24c26" event={"ID":"d6bce77f-3e2f-40af-8109-b52446e53c4a","Type":"ContainerStarted","Data":"85dbf375149b0f83fa2c3d6f689d8d3f3d8fc10780889c4a8f2020c26fa74a76"} Jul 24 17:44:34.224592 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:34.224543 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-24c26" podStartSLOduration=34.024286451 podStartE2EDuration="46.224528025s" totalImagesPullingTime="12.200241574s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 17:43:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:44:34.222699664 +0000 UTC m=+99.262493704" watchObservedRunningTime="2026-07-24 17:44:34.224528025 +0000 UTC m=+99.264322065" Jul 24 17:44:38.335737 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:38.335699 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-658f8b44f6-hkqlq"] Jul 24 17:44:59.295412 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:59.295376 2559 generic.go:356] "Generic (PLEG): container finished" podID="aebd90e8-411f-4c87-bed7-eb3a1ca82458" containerID="6f098caaefa9fbaff882ca37621452bfe73c3cd9050b25f84f320a865a821239" exitCode=0 Jul 24 17:44:59.295843 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:59.295419 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" event={"ID":"aebd90e8-411f-4c87-bed7-eb3a1ca82458","Type":"ContainerDied","Data":"6f098caaefa9fbaff882ca37621452bfe73c3cd9050b25f84f320a865a821239"} Jul 24 17:44:59.295843 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:44:59.295713 2559 scope.go:117] "RemoveContainer" containerID="6f098caaefa9fbaff882ca37621452bfe73c3cd9050b25f84f320a865a821239" Jul 24 17:45:00.301307 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:00.301272 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lk7dt" event={"ID":"aebd90e8-411f-4c87-bed7-eb3a1ca82458","Type":"ContainerStarted","Data":"d9b2f37a2be42c69f1488691deb6cbb03b63ab7d395e1841ea39b06d2ac5fcdb"} Jul 24 17:45:02.397680 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:02.397637 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" podUID="88676e21-31ea-4fdf-963d-04c2e0fb7c75" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 24 17:45:03.355537 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:03.355494 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" podUID="fa94510b-f873-4c29-8b7b-668248d6e4cb" containerName="registry" containerID="cri-o://265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886" gracePeriod=30 Jul 24 17:45:04.147601 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.147576 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:45:04.246054 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.245968 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-installation-pull-secrets\") pod \"fa94510b-f873-4c29-8b7b-668248d6e4cb\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " Jul 24 17:45:04.246054 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.246022 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sj448\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-kube-api-access-sj448\") pod \"fa94510b-f873-4c29-8b7b-668248d6e4cb\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " Jul 24 17:45:04.246054 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.246048 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-bound-sa-token\") pod \"fa94510b-f873-4c29-8b7b-668248d6e4cb\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " Jul 24 17:45:04.246338 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.246078 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa94510b-f873-4c29-8b7b-668248d6e4cb-ca-trust-extracted\") pod \"fa94510b-f873-4c29-8b7b-668248d6e4cb\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " Jul 24 17:45:04.246338 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.246111 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") pod \"fa94510b-f873-4c29-8b7b-668248d6e4cb\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " Jul 24 17:45:04.246338 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.246127 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-trusted-ca\") pod \"fa94510b-f873-4c29-8b7b-668248d6e4cb\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " Jul 24 17:45:04.246338 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.246152 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-certificates\") pod \"fa94510b-f873-4c29-8b7b-668248d6e4cb\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " Jul 24 17:45:04.246338 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.246169 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-image-registry-private-configuration\") pod \"fa94510b-f873-4c29-8b7b-668248d6e4cb\" (UID: \"fa94510b-f873-4c29-8b7b-668248d6e4cb\") " Jul 24 17:45:04.246661 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.246624 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "fa94510b-f873-4c29-8b7b-668248d6e4cb" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:45:04.247303 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.247267 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "fa94510b-f873-4c29-8b7b-668248d6e4cb" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 17:45:04.248729 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.248683 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "fa94510b-f873-4c29-8b7b-668248d6e4cb" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:45:04.248729 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.248689 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "fa94510b-f873-4c29-8b7b-668248d6e4cb" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:45:04.248912 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.248809 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "fa94510b-f873-4c29-8b7b-668248d6e4cb" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 17:45:04.249014 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.248992 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-kube-api-access-sj448" (OuterVolumeSpecName: "kube-api-access-sj448") pod "fa94510b-f873-4c29-8b7b-668248d6e4cb" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb"). InnerVolumeSpecName "kube-api-access-sj448". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:45:04.249345 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.249326 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "fa94510b-f873-4c29-8b7b-668248d6e4cb" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:45:04.255148 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.255118 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa94510b-f873-4c29-8b7b-668248d6e4cb-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "fa94510b-f873-4c29-8b7b-668248d6e4cb" (UID: "fa94510b-f873-4c29-8b7b-668248d6e4cb"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:45:04.317079 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.317046 2559 generic.go:356] "Generic (PLEG): container finished" podID="fa94510b-f873-4c29-8b7b-668248d6e4cb" containerID="265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886" exitCode=0 Jul 24 17:45:04.317254 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.317084 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" Jul 24 17:45:04.317254 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.317103 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" event={"ID":"fa94510b-f873-4c29-8b7b-668248d6e4cb","Type":"ContainerDied","Data":"265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886"} Jul 24 17:45:04.317254 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.317153 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-658f8b44f6-hkqlq" event={"ID":"fa94510b-f873-4c29-8b7b-668248d6e4cb","Type":"ContainerDied","Data":"c4ced713aa5dc4f41675214b9a8a9a69e3616f9a9a75d917f65aa0e6572529a2"} Jul 24 17:45:04.317254 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.317179 2559 scope.go:117] "RemoveContainer" containerID="265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886" Jul 24 17:45:04.326025 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.326004 2559 scope.go:117] "RemoveContainer" containerID="265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886" Jul 24 17:45:04.326277 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:45:04.326255 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886\": container with ID starting with 265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886 not found: ID does not exist" containerID="265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886" Jul 24 17:45:04.326329 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.326286 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886"} err="failed to get container status \"265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886\": rpc error: code = NotFound desc = could not find container \"265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886\": container with ID starting with 265967171e4082b16c71a25c08a370d63dbd6de842f4ecb930c3022123d44886 not found: ID does not exist" Jul 24 17:45:04.338982 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.338932 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-658f8b44f6-hkqlq"] Jul 24 17:45:04.345398 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.345367 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-658f8b44f6-hkqlq"] Jul 24 17:45:04.347304 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.347285 2559 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-installation-pull-secrets\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:45:04.347358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.347307 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sj448\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-kube-api-access-sj448\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:45:04.347358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.347318 2559 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-bound-sa-token\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:45:04.347358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.347326 2559 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/fa94510b-f873-4c29-8b7b-668248d6e4cb-ca-trust-extracted\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:45:04.347358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.347334 2559 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-tls\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:45:04.347358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.347342 2559 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-trusted-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:45:04.347358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.347349 2559 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/fa94510b-f873-4c29-8b7b-668248d6e4cb-registry-certificates\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:45:04.347358 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:04.347357 2559 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/fa94510b-f873-4c29-8b7b-668248d6e4cb-image-registry-private-configuration\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:45:05.254059 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:05.254024 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:45:05.256379 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:05.256354 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a-metrics-certs\") pod \"network-metrics-daemon-5pvds\" (UID: \"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a\") " pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:45:05.460752 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:05.460720 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-pdhgn\"" Jul 24 17:45:05.468935 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:05.468914 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-5pvds" Jul 24 17:45:05.493836 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:05.493772 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa94510b-f873-4c29-8b7b-668248d6e4cb" path="/var/lib/kubelet/pods/fa94510b-f873-4c29-8b7b-668248d6e4cb/volumes" Jul 24 17:45:05.613585 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:05.613441 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-5pvds"] Jul 24 17:45:05.616107 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:45:05.616083 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5b05cd_5eb8_40cf_94a5_8dc0113ae48a.slice/crio-3313f563dd512bac2a5feceaa2cc6abf8c04528a93265577e152411bdfc6ca5b WatchSource:0}: Error finding container 3313f563dd512bac2a5feceaa2cc6abf8c04528a93265577e152411bdfc6ca5b: Status 404 returned error can't find the container with id 3313f563dd512bac2a5feceaa2cc6abf8c04528a93265577e152411bdfc6ca5b Jul 24 17:45:06.327229 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:06.327194 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pvds" event={"ID":"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a","Type":"ContainerStarted","Data":"3313f563dd512bac2a5feceaa2cc6abf8c04528a93265577e152411bdfc6ca5b"} Jul 24 17:45:07.332697 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:07.332659 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pvds" event={"ID":"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a","Type":"ContainerStarted","Data":"16c8b00402234627ea53bb6265fa317c14ded942606c6edf943427b8ee77baf3"} Jul 24 17:45:07.333191 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:07.332719 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-5pvds" event={"ID":"1c5b05cd-5eb8-40cf-94a5-8dc0113ae48a","Type":"ContainerStarted","Data":"262ce9caa66bbaa91207963211e64d378cbebfb88aa5456d379d22d55c4e79cb"} Jul 24 17:45:07.350707 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:07.350654 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-5pvds" podStartSLOduration=131.469819589 podStartE2EDuration="2m12.350638154s" totalImagesPullingTime="880.818565ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:42:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:45:07.3494584 +0000 UTC m=+132.389252442" watchObservedRunningTime="2026-07-24 17:45:07.350638154 +0000 UTC m=+132.390432195" Jul 24 17:45:12.398018 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:12.397979 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" podUID="88676e21-31ea-4fdf-963d-04c2e0fb7c75" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 24 17:45:19.377453 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:19.377422 2559 generic.go:356] "Generic (PLEG): container finished" podID="6f5e6ac9-6056-4018-a77e-33ac5e69cf72" containerID="c5a96f799c5a7902342023e5368aee0832df1c3268731297cd14c3b7bc10ca51" exitCode=0 Jul 24 17:45:19.377928 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:19.377506 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" event={"ID":"6f5e6ac9-6056-4018-a77e-33ac5e69cf72","Type":"ContainerDied","Data":"c5a96f799c5a7902342023e5368aee0832df1c3268731297cd14c3b7bc10ca51"} Jul 24 17:45:19.377928 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:19.377923 2559 scope.go:117] "RemoveContainer" containerID="c5a96f799c5a7902342023e5368aee0832df1c3268731297cd14c3b7bc10ca51" Jul 24 17:45:19.379036 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:19.379014 2559 generic.go:356] "Generic (PLEG): container finished" podID="b9c8447a-8f08-4d76-a7d3-103f60b7e495" containerID="5c137d650aca39918359d5a1c2d23527676a160ee28a1395c4af43a273908258" exitCode=0 Jul 24 17:45:19.379134 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:19.379081 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-k2526" event={"ID":"b9c8447a-8f08-4d76-a7d3-103f60b7e495","Type":"ContainerDied","Data":"5c137d650aca39918359d5a1c2d23527676a160ee28a1395c4af43a273908258"} Jul 24 17:45:19.379377 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:19.379363 2559 scope.go:117] "RemoveContainer" containerID="5c137d650aca39918359d5a1c2d23527676a160ee28a1395c4af43a273908258" Jul 24 17:45:20.384718 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:20.384683 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5885fdb759-k2526" event={"ID":"b9c8447a-8f08-4d76-a7d3-103f60b7e495","Type":"ContainerStarted","Data":"a62f3cf2ccff6ce2b5fbf0e8f68218bc6550124e9318a584bdf79d5ab3b466d4"} Jul 24 17:45:20.386477 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:20.386455 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf" event={"ID":"6f5e6ac9-6056-4018-a77e-33ac5e69cf72","Type":"ContainerStarted","Data":"b65f11cd3867c8b6939b1c70d13fbac2c0228906b02ec109a7022a3116f3d6a8"} Jul 24 17:45:22.397908 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:22.397868 2559 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" podUID="88676e21-31ea-4fdf-963d-04c2e0fb7c75" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 24 17:45:22.398359 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:22.397938 2559 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" Jul 24 17:45:22.398420 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:22.398403 2559 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"c0810f69a676d54c6f8d8a4d32797d3dee002049259c2e7bdcd0fee84fe4d7e3"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 24 17:45:22.398463 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:22.398440 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" podUID="88676e21-31ea-4fdf-963d-04c2e0fb7c75" containerName="service-proxy" containerID="cri-o://c0810f69a676d54c6f8d8a4d32797d3dee002049259c2e7bdcd0fee84fe4d7e3" gracePeriod=30 Jul 24 17:45:23.400580 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:23.400548 2559 generic.go:356] "Generic (PLEG): container finished" podID="88676e21-31ea-4fdf-963d-04c2e0fb7c75" containerID="c0810f69a676d54c6f8d8a4d32797d3dee002049259c2e7bdcd0fee84fe4d7e3" exitCode=2 Jul 24 17:45:23.400990 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:23.400598 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" event={"ID":"88676e21-31ea-4fdf-963d-04c2e0fb7c75","Type":"ContainerDied","Data":"c0810f69a676d54c6f8d8a4d32797d3dee002049259c2e7bdcd0fee84fe4d7e3"} Jul 24 17:45:23.400990 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:45:23.400631 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79b4ffcdcc-hzf45" event={"ID":"88676e21-31ea-4fdf-963d-04c2e0fb7c75","Type":"ContainerStarted","Data":"a721e4322ba68229bffaa808eff8a28d307f81bcd0eafbe20dcd44eee4e8025c"} Jul 24 17:47:24.736705 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.736625 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp"] Jul 24 17:47:24.737187 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.736914 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa94510b-f873-4c29-8b7b-668248d6e4cb" containerName="registry" Jul 24 17:47:24.737187 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.736925 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa94510b-f873-4c29-8b7b-668248d6e4cb" containerName="registry" Jul 24 17:47:24.737187 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.736989 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa94510b-f873-4c29-8b7b-668248d6e4cb" containerName="registry" Jul 24 17:47:24.739980 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.739962 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.743406 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.743386 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 24 17:47:24.743996 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.743975 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 24 17:47:24.744102 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.744009 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 24 17:47:24.744102 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.744056 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-vnvl4\"" Jul 24 17:47:24.744102 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.744063 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 24 17:47:24.775985 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.775954 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp"] Jul 24 17:47:24.838445 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.838412 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08178467-f23e-4195-b654-953476d882dd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.838618 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.838449 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkcrh\" (UniqueName: \"kubernetes.io/projected/08178467-f23e-4195-b654-953476d882dd-kube-api-access-fkcrh\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.838618 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.838472 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08178467-f23e-4195-b654-953476d882dd-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.838618 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.838571 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08178467-f23e-4195-b654-953476d882dd-manifests\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.838618 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.838609 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08178467-f23e-4195-b654-953476d882dd-tmp\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.939322 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.939287 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08178467-f23e-4195-b654-953476d882dd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.939322 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.939323 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkcrh\" (UniqueName: \"kubernetes.io/projected/08178467-f23e-4195-b654-953476d882dd-kube-api-access-fkcrh\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.939540 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.939340 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08178467-f23e-4195-b654-953476d882dd-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.939540 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.939381 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08178467-f23e-4195-b654-953476d882dd-manifests\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.939540 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.939403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08178467-f23e-4195-b654-953476d882dd-tmp\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.939989 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.939963 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/08178467-f23e-4195-b654-953476d882dd-manifests\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.941921 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.941894 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/08178467-f23e-4195-b654-953476d882dd-tmp\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.942051 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.942002 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/08178467-f23e-4195-b654-953476d882dd-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.942194 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.942173 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/08178467-f23e-4195-b654-953476d882dd-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:24.947622 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:24.947604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkcrh\" (UniqueName: \"kubernetes.io/projected/08178467-f23e-4195-b654-953476d882dd-kube-api-access-fkcrh\") pod \"opendatahub-operator-controller-manager-dbf4bd545-2tbcp\" (UID: \"08178467-f23e-4195-b654-953476d882dd\") " pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:25.049539 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:25.049457 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:25.191926 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:25.191906 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp"] Jul 24 17:47:25.194570 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:47:25.194542 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08178467_f23e_4195_b654_953476d882dd.slice/crio-06a11fa90eee0e52c6cc23f99a89bf0087d09cc15c324b77a8fd17a5dc47bdda WatchSource:0}: Error finding container 06a11fa90eee0e52c6cc23f99a89bf0087d09cc15c324b77a8fd17a5dc47bdda: Status 404 returned error can't find the container with id 06a11fa90eee0e52c6cc23f99a89bf0087d09cc15c324b77a8fd17a5dc47bdda Jul 24 17:47:25.875334 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:25.875260 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" event={"ID":"08178467-f23e-4195-b654-953476d882dd","Type":"ContainerStarted","Data":"06a11fa90eee0e52c6cc23f99a89bf0087d09cc15c324b77a8fd17a5dc47bdda"} Jul 24 17:47:28.891329 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:28.891291 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" event={"ID":"08178467-f23e-4195-b654-953476d882dd","Type":"ContainerStarted","Data":"962b83b433919be740d51ce6c1d9db3af0a2264b766097afd44e3e5a0f0b2214"} Jul 24 17:47:29.899022 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:29.898988 2559 generic.go:356] "Generic (PLEG): container finished" podID="08178467-f23e-4195-b654-953476d882dd" containerID="962b83b433919be740d51ce6c1d9db3af0a2264b766097afd44e3e5a0f0b2214" exitCode=0 Jul 24 17:47:29.899423 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:29.899078 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" event={"ID":"08178467-f23e-4195-b654-953476d882dd","Type":"ContainerDied","Data":"962b83b433919be740d51ce6c1d9db3af0a2264b766097afd44e3e5a0f0b2214"} Jul 24 17:47:30.293097 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:47:30.293065 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08178467_f23e_4195_b654_953476d882dd.slice/crio-faa1b686a563d5dd2076d56c36f7e1aee21b4d5f54c2e44194e53cd7b2fb7139.scope\": RecentStats: unable to find data in memory cache]" Jul 24 17:47:30.905333 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:30.905292 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" event={"ID":"08178467-f23e-4195-b654-953476d882dd","Type":"ContainerStarted","Data":"faa1b686a563d5dd2076d56c36f7e1aee21b4d5f54c2e44194e53cd7b2fb7139"} Jul 24 17:47:30.905715 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:30.905421 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:30.954218 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:30.954168 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" podStartSLOduration=2.673003499 podStartE2EDuration="6.954151233s" totalImagesPullingTime="3.255171501s" totalInitContainerRuntime="1.025976233s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:30.95263309 +0000 UTC m=+275.992427131" watchObservedRunningTime="2026-07-24 17:47:30.954151233 +0000 UTC m=+275.993945273" Jul 24 17:47:41.912310 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:41.912283 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dbf4bd545-2tbcp" Jul 24 17:47:51.249686 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.249651 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k"] Jul 24 17:47:51.252943 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.252926 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.255642 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.255496 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 24 17:47:51.255642 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.255577 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-7mhq7\"" Jul 24 17:47:51.267568 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.267545 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k"] Jul 24 17:47:51.335419 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335389 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.335419 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335450 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjff7\" (UniqueName: \"kubernetes.io/projected/58092870-06db-4d0f-ba90-12e6ecd13d95-kube-api-access-hjff7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.335701 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335540 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.335701 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335580 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.335701 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.335701 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335653 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58092870-06db-4d0f-ba90-12e6ecd13d95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.335701 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335674 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.335906 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335743 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.335906 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.335800 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.436895 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.436861 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58092870-06db-4d0f-ba90-12e6ecd13d95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.436895 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.436896 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.436926 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.436947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.436973 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437033 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjff7\" (UniqueName: \"kubernetes.io/projected/58092870-06db-4d0f-ba90-12e6ecd13d95-kube-api-access-hjff7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437063 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437097 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437119 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437464 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437330 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437464 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437389 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437577 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437472 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437703 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437683 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.437765 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.437697 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58092870-06db-4d0f-ba90-12e6ecd13d95-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.439547 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.439522 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.439696 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.439676 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.451168 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.451133 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58092870-06db-4d0f-ba90-12e6ecd13d95-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.451390 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.451371 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjff7\" (UniqueName: \"kubernetes.io/projected/58092870-06db-4d0f-ba90-12e6ecd13d95-kube-api-access-hjff7\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k\" (UID: \"58092870-06db-4d0f-ba90-12e6ecd13d95\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.562892 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.562863 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:51.707163 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.707143 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k"] Jul 24 17:47:51.709258 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:47:51.709234 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58092870_06db_4d0f_ba90_12e6ecd13d95.slice/crio-6cd22d57477e818c6419ce336fb033ac2a53dac4bfcd2d0572d71d924ac442bf WatchSource:0}: Error finding container 6cd22d57477e818c6419ce336fb033ac2a53dac4bfcd2d0572d71d924ac442bf: Status 404 returned error can't find the container with id 6cd22d57477e818c6419ce336fb033ac2a53dac4bfcd2d0572d71d924ac442bf Jul 24 17:47:51.992150 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:51.992067 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" event={"ID":"58092870-06db-4d0f-ba90-12e6ecd13d95","Type":"ContainerStarted","Data":"6cd22d57477e818c6419ce336fb033ac2a53dac4bfcd2d0572d71d924ac442bf"} Jul 24 17:47:54.037982 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.037951 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-9f897544f-crvdf"] Jul 24 17:47:54.044947 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.044921 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.047899 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.047864 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 24 17:47:54.047899 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.047864 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-dbqjz\"" Jul 24 17:47:54.049145 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.048992 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 24 17:47:54.049145 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.048992 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 24 17:47:54.049145 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.049039 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 24 17:47:54.049375 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.049287 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 24 17:47:54.053020 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.052977 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9f897544f-crvdf"] Jul 24 17:47:54.159710 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.159667 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd9785f3-bf18-4643-8a7f-8961b0aecd58-manager-config\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.159710 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.159712 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd9785f3-bf18-4643-8a7f-8961b0aecd58-cert\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.159966 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.159737 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd9785f3-bf18-4643-8a7f-8961b0aecd58-metrics-cert\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.159966 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.159874 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75wdb\" (UniqueName: \"kubernetes.io/projected/bd9785f3-bf18-4643-8a7f-8961b0aecd58-kube-api-access-75wdb\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.260609 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.260567 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd9785f3-bf18-4643-8a7f-8961b0aecd58-manager-config\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.260816 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.260621 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd9785f3-bf18-4643-8a7f-8961b0aecd58-cert\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.260816 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.260646 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd9785f3-bf18-4643-8a7f-8961b0aecd58-metrics-cert\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.260816 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.260742 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75wdb\" (UniqueName: \"kubernetes.io/projected/bd9785f3-bf18-4643-8a7f-8961b0aecd58-kube-api-access-75wdb\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.261526 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.261446 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/bd9785f3-bf18-4643-8a7f-8961b0aecd58-manager-config\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.263395 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.263376 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bd9785f3-bf18-4643-8a7f-8961b0aecd58-cert\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.263554 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.263529 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/bd9785f3-bf18-4643-8a7f-8961b0aecd58-metrics-cert\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.275359 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.275326 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-75wdb\" (UniqueName: \"kubernetes.io/projected/bd9785f3-bf18-4643-8a7f-8961b0aecd58-kube-api-access-75wdb\") pod \"lws-controller-manager-9f897544f-crvdf\" (UID: \"bd9785f3-bf18-4643-8a7f-8961b0aecd58\") " pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.357872 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.357828 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:54.465353 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.465306 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:47:54.465472 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.465409 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:47:54.465472 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.465455 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:47:54.535880 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:54.535856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-9f897544f-crvdf"] Jul 24 17:47:54.537368 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:47:54.537339 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd9785f3_bf18_4643_8a7f_8961b0aecd58.slice/crio-04824bc7a0c091664307d2b9a7afad0da743c685eb89292d964e99ea99240a33 WatchSource:0}: Error finding container 04824bc7a0c091664307d2b9a7afad0da743c685eb89292d964e99ea99240a33: Status 404 returned error can't find the container with id 04824bc7a0c091664307d2b9a7afad0da743c685eb89292d964e99ea99240a33 Jul 24 17:47:55.008364 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:55.008316 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" event={"ID":"58092870-06db-4d0f-ba90-12e6ecd13d95","Type":"ContainerStarted","Data":"13f2ac37d8d0a862e6db59c95eb10ab5846a913aeec02dddba4e40f552d7f9a1"} Jul 24 17:47:55.009623 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:55.009579 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" event={"ID":"bd9785f3-bf18-4643-8a7f-8961b0aecd58","Type":"ContainerStarted","Data":"04824bc7a0c091664307d2b9a7afad0da743c685eb89292d964e99ea99240a33"} Jul 24 17:47:55.062214 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:55.062157 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" podStartSLOduration=1.308588739 podStartE2EDuration="4.062140032s" totalImagesPullingTime="2.753551293s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:55.050076113 +0000 UTC m=+300.089870145" watchObservedRunningTime="2026-07-24 17:47:55.062140032 +0000 UTC m=+300.101934072" Jul 24 17:47:55.393709 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:55.393678 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:47:55.403848 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:55.403818 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:47:55.420485 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:55.420460 2559 kubelet.go:1648] "Image garbage collection succeeded" Jul 24 17:47:55.563665 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:55.563630 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:55.575277 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:55.569718 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:56.015109 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.015079 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:56.016186 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.016161 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k" Jul 24 17:47:56.433567 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.433529 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr"] Jul 24 17:47:56.442205 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.442183 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:47:56.444868 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.444845 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-66fnh\"" Jul 24 17:47:56.445019 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.444998 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 24 17:47:56.454814 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.454771 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn"] Jul 24 17:47:56.458232 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.458214 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:56.465589 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.465564 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 24 17:47:56.465712 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.465597 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-vhxt9\"" Jul 24 17:47:56.470024 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.469999 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr"] Jul 24 17:47:56.484278 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.484244 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxqrg\" (UniqueName: \"kubernetes.io/projected/6c4537aa-c3c0-4d96-ab42-0a16dca59c48-kube-api-access-wxqrg\") pod \"kserve-controller-manager-d6f4b65b8-xdbrr\" (UID: \"6c4537aa-c3c0-4d96-ab42-0a16dca59c48\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:47:56.484433 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.484330 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c4537aa-c3c0-4d96-ab42-0a16dca59c48-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdbrr\" (UID: \"6c4537aa-c3c0-4d96-ab42-0a16dca59c48\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:47:56.492376 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.492344 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn"] Jul 24 17:47:56.585436 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.585406 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vng49\" (UniqueName: \"kubernetes.io/projected/46c58efc-9286-443c-abfe-e03a81664413-kube-api-access-vng49\") pod \"llmisvc-controller-manager-777c6bd7d5-pvvnn\" (UID: \"46c58efc-9286-443c-abfe-e03a81664413\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:56.585436 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.585444 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxqrg\" (UniqueName: \"kubernetes.io/projected/6c4537aa-c3c0-4d96-ab42-0a16dca59c48-kube-api-access-wxqrg\") pod \"kserve-controller-manager-d6f4b65b8-xdbrr\" (UID: \"6c4537aa-c3c0-4d96-ab42-0a16dca59c48\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:47:56.585702 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.585474 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46c58efc-9286-443c-abfe-e03a81664413-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pvvnn\" (UID: \"46c58efc-9286-443c-abfe-e03a81664413\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:56.585766 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.585707 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c4537aa-c3c0-4d96-ab42-0a16dca59c48-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdbrr\" (UID: \"6c4537aa-c3c0-4d96-ab42-0a16dca59c48\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:47:56.588517 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.588493 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c4537aa-c3c0-4d96-ab42-0a16dca59c48-cert\") pod \"kserve-controller-manager-d6f4b65b8-xdbrr\" (UID: \"6c4537aa-c3c0-4d96-ab42-0a16dca59c48\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:47:56.594122 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.594092 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxqrg\" (UniqueName: \"kubernetes.io/projected/6c4537aa-c3c0-4d96-ab42-0a16dca59c48-kube-api-access-wxqrg\") pod \"kserve-controller-manager-d6f4b65b8-xdbrr\" (UID: \"6c4537aa-c3c0-4d96-ab42-0a16dca59c48\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:47:56.686256 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.686163 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vng49\" (UniqueName: \"kubernetes.io/projected/46c58efc-9286-443c-abfe-e03a81664413-kube-api-access-vng49\") pod \"llmisvc-controller-manager-777c6bd7d5-pvvnn\" (UID: \"46c58efc-9286-443c-abfe-e03a81664413\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:56.686256 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.686207 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46c58efc-9286-443c-abfe-e03a81664413-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pvvnn\" (UID: \"46c58efc-9286-443c-abfe-e03a81664413\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:56.686479 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:47:56.686353 2559 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found Jul 24 17:47:56.686479 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:47:56.686433 2559 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/46c58efc-9286-443c-abfe-e03a81664413-cert podName:46c58efc-9286-443c-abfe-e03a81664413 nodeName:}" failed. No retries permitted until 2026-07-24 17:47:57.186412534 +0000 UTC m=+302.226206557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/46c58efc-9286-443c-abfe-e03a81664413-cert") pod "llmisvc-controller-manager-777c6bd7d5-pvvnn" (UID: "46c58efc-9286-443c-abfe-e03a81664413") : secret "llmisvc-webhook-server-cert" not found Jul 24 17:47:56.698678 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.698647 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vng49\" (UniqueName: \"kubernetes.io/projected/46c58efc-9286-443c-abfe-e03a81664413-kube-api-access-vng49\") pod \"llmisvc-controller-manager-777c6bd7d5-pvvnn\" (UID: \"46c58efc-9286-443c-abfe-e03a81664413\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:56.752998 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:56.752964 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:47:57.190102 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:57.190071 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46c58efc-9286-443c-abfe-e03a81664413-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pvvnn\" (UID: \"46c58efc-9286-443c-abfe-e03a81664413\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:57.193134 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:57.193106 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/46c58efc-9286-443c-abfe-e03a81664413-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-pvvnn\" (UID: \"46c58efc-9286-443c-abfe-e03a81664413\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:57.209631 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:57.209598 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr"] Jul 24 17:47:57.211508 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:47:57.211479 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c4537aa_c3c0_4d96_ab42_0a16dca59c48.slice/crio-329ad5ddb9cb08bea430aad9859251947ceb027a1663b8eac5bec1363b308d6d WatchSource:0}: Error finding container 329ad5ddb9cb08bea430aad9859251947ceb027a1663b8eac5bec1363b308d6d: Status 404 returned error can't find the container with id 329ad5ddb9cb08bea430aad9859251947ceb027a1663b8eac5bec1363b308d6d Jul 24 17:47:57.212895 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:57.212879 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 17:47:57.369089 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:57.368993 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:47:57.524624 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:57.524592 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn"] Jul 24 17:47:57.527623 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:47:57.527595 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod46c58efc_9286_443c_abfe_e03a81664413.slice/crio-f7a9bbb1bbd55c8dc9ab812795164db0c540fadd35dbd0b1b9fa0ebf06008025 WatchSource:0}: Error finding container f7a9bbb1bbd55c8dc9ab812795164db0c540fadd35dbd0b1b9fa0ebf06008025: Status 404 returned error can't find the container with id f7a9bbb1bbd55c8dc9ab812795164db0c540fadd35dbd0b1b9fa0ebf06008025 Jul 24 17:47:58.027835 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:58.027776 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" event={"ID":"6c4537aa-c3c0-4d96-ab42-0a16dca59c48","Type":"ContainerStarted","Data":"329ad5ddb9cb08bea430aad9859251947ceb027a1663b8eac5bec1363b308d6d"} Jul 24 17:47:58.029966 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:58.029661 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" event={"ID":"bd9785f3-bf18-4643-8a7f-8961b0aecd58","Type":"ContainerStarted","Data":"6a4395db7fdd89096f7f736e95a5ddb61db6808ffe720ccfbc21aba6a1d5704b"} Jul 24 17:47:58.029966 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:58.029823 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:47:58.031248 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:58.031227 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" event={"ID":"46c58efc-9286-443c-abfe-e03a81664413","Type":"ContainerStarted","Data":"f7a9bbb1bbd55c8dc9ab812795164db0c540fadd35dbd0b1b9fa0ebf06008025"} Jul 24 17:47:58.065564 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:47:58.065503 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" podStartSLOduration=1.4624826739999999 podStartE2EDuration="4.065482029s" totalImagesPullingTime="2.602999355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:47:58.063924751 +0000 UTC m=+303.103718783" watchObservedRunningTime="2026-07-24 17:47:58.065482029 +0000 UTC m=+303.105276070" Jul 24 17:48:00.045220 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:00.045182 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" event={"ID":"6c4537aa-c3c0-4d96-ab42-0a16dca59c48","Type":"ContainerStarted","Data":"a58e96d97d0cba88eead3d0f0ae143aff5e670879d330ff030dc02cdbd788ec3"} Jul 24 17:48:00.045714 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:00.045343 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:48:00.070796 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:00.070732 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" podStartSLOduration=1.7348055489999998 podStartE2EDuration="4.070711472s" totalImagesPullingTime="2.335905923s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:00.070376671 +0000 UTC m=+305.110170713" watchObservedRunningTime="2026-07-24 17:48:00.070711472 +0000 UTC m=+305.110505514" Jul 24 17:48:01.051048 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:01.051010 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" event={"ID":"46c58efc-9286-443c-abfe-e03a81664413","Type":"ContainerStarted","Data":"fa6f951ccfe6c0f873c6ff2353c8453acd076bb27aba0417a8357dda0d905135"} Jul 24 17:48:01.083765 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:01.083715 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" podStartSLOduration=2.451408385 podStartE2EDuration="5.083701546s" totalImagesPullingTime="2.632293161s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:47:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:48:01.083548308 +0000 UTC m=+306.123342348" watchObservedRunningTime="2026-07-24 17:48:01.083701546 +0000 UTC m=+306.123495586" Jul 24 17:48:02.054862 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:02.054829 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:48:09.039576 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:09.039547 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-9f897544f-crvdf" Jul 24 17:48:31.056425 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:31.056396 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-xdbrr" Jul 24 17:48:33.061411 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:48:33.061381 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-pvvnn" Jul 24 17:49:28.862119 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:28.862085 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf"] Jul 24 17:49:28.865338 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:28.865322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:28.868641 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:28.868619 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-4znsw\"" Jul 24 17:49:28.868851 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:28.868839 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 17:49:28.874453 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:28.874436 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 17:49:28.904551 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:28.904522 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf"] Jul 24 17:49:28.980141 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:28.980109 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcszh\" (UniqueName: \"kubernetes.io/projected/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-kube-api-access-bcszh\") pod \"kuadrant-operator-controller-manager-96db98c47-2mscf\" (UID: \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:28.980324 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:28.980157 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-2mscf\" (UID: \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:29.081261 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.081228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcszh\" (UniqueName: \"kubernetes.io/projected/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-kube-api-access-bcszh\") pod \"kuadrant-operator-controller-manager-96db98c47-2mscf\" (UID: \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:29.081439 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.081273 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-2mscf\" (UID: \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:29.081596 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.081581 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-96db98c47-2mscf\" (UID: \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:29.097210 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.097184 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcszh\" (UniqueName: \"kubernetes.io/projected/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-kube-api-access-bcszh\") pod \"kuadrant-operator-controller-manager-96db98c47-2mscf\" (UID: \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:29.123441 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.123359 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf"] Jul 24 17:49:29.123637 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.123626 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:29.290962 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.287423 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf"] Jul 24 17:49:29.294219 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:49:29.294192 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2d6eff1_80d8_4a08_a82b_d928c59db6e1.slice/crio-88e872ec5f4a09c3aff895c055f4587cb4bd2bf040106c1e4c5771b2a598a78d WatchSource:0}: Error finding container 88e872ec5f4a09c3aff895c055f4587cb4bd2bf040106c1e4c5771b2a598a78d: Status 404 returned error can't find the container with id 88e872ec5f4a09c3aff895c055f4587cb4bd2bf040106c1e4c5771b2a598a78d Jul 24 17:49:29.418667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.418586 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" event={"ID":"a2d6eff1-80d8-4a08-a82b-d928c59db6e1","Type":"ContainerStarted","Data":"88e872ec5f4a09c3aff895c055f4587cb4bd2bf040106c1e4c5771b2a598a78d"} Jul 24 17:49:29.529846 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.529779 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf"] Jul 24 17:49:29.542322 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:29.542279 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf"] Jul 24 17:49:33.444070 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:33.444012 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" podUID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" containerName="manager" containerID="cri-o://c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b" gracePeriod=2 Jul 24 17:49:33.688102 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:33.688076 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:33.720257 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:33.720181 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-extensions-socket-volume\") pod \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\" (UID: \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\") " Jul 24 17:49:33.720257 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:33.720249 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcszh\" (UniqueName: \"kubernetes.io/projected/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-kube-api-access-bcszh\") pod \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\" (UID: \"a2d6eff1-80d8-4a08-a82b-d928c59db6e1\") " Jul 24 17:49:33.720451 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:33.720428 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "a2d6eff1-80d8-4a08-a82b-d928c59db6e1" (UID: "a2d6eff1-80d8-4a08-a82b-d928c59db6e1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 17:49:33.720547 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:33.720515 2559 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-extensions-socket-volume\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:49:33.722667 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:33.722639 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-kube-api-access-bcszh" (OuterVolumeSpecName: "kube-api-access-bcszh") pod "a2d6eff1-80d8-4a08-a82b-d928c59db6e1" (UID: "a2d6eff1-80d8-4a08-a82b-d928c59db6e1"). InnerVolumeSpecName "kube-api-access-bcszh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 17:49:33.821496 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:33.821462 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcszh\" (UniqueName: \"kubernetes.io/projected/a2d6eff1-80d8-4a08-a82b-d928c59db6e1-kube-api-access-bcszh\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 17:49:34.449579 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:34.449542 2559 generic.go:356] "Generic (PLEG): container finished" podID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" containerID="c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b" exitCode=2 Jul 24 17:49:34.450190 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:34.449590 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:34.450190 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:34.449623 2559 scope.go:117] "RemoveContainer" containerID="c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b" Jul 24 17:49:34.452150 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:49:34.452125 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-2mscf\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:34.458094 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:34.458079 2559 scope.go:117] "RemoveContainer" containerID="c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b" Jul 24 17:49:34.458351 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:49:34.458336 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b\": container with ID starting with c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b not found: ID does not exist" containerID="c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b" Jul 24 17:49:34.458389 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:34.458362 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b"} err="failed to get container status \"c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b\": rpc error: code = NotFound desc = could not find container \"c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b\": container with ID starting with c2ec0abb80b1fdb88ae4b2adb9439ea9dec66e27c81f688de080043e4c83946b not found: ID does not exist" Jul 24 17:49:34.459932 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:49:34.459910 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-2mscf\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:35.493184 ip-10-0-141-191 kubenswrapper[2559]: E0724 17:49:35.493098 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-96db98c47-2mscf\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" pod="kuadrant-system/kuadrant-operator-controller-manager-96db98c47-2mscf" Jul 24 17:49:35.494488 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:35.494465 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" path="/var/lib/kubelet/pods/a2d6eff1-80d8-4a08-a82b-d928c59db6e1/volumes" Jul 24 17:49:59.211275 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.211244 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt"] Jul 24 17:49:59.211711 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.211570 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" containerName="manager" Jul 24 17:49:59.211711 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.211580 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" containerName="manager" Jul 24 17:49:59.211711 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.211644 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="a2d6eff1-80d8-4a08-a82b-d928c59db6e1" containerName="manager" Jul 24 17:49:59.215730 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.215716 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.218374 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.218351 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-cxrxp\"" Jul 24 17:49:59.225843 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.225820 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt"] Jul 24 17:49:59.340151 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340116 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.340151 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340154 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg7qq\" (UniqueName: \"kubernetes.io/projected/d756e0ed-56b3-473c-8087-353d2eabd7b8-kube-api-access-mg7qq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.340393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340180 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.340393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340240 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.340393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340286 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.340393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340309 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.340393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340332 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.340393 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.340593 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.340402 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d756e0ed-56b3-473c-8087-353d2eabd7b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.440807 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.440757 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d756e0ed-56b3-473c-8087-353d2eabd7b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.440991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.440834 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.440991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.440865 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg7qq\" (UniqueName: \"kubernetes.io/projected/d756e0ed-56b3-473c-8087-353d2eabd7b8-kube-api-access-mg7qq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.440991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.440889 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.440991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.440909 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.440991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.440929 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.440991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.440956 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.440991 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.440980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.441350 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.441025 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.441350 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.441317 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.441462 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.441361 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.441812 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.441766 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d756e0ed-56b3-473c-8087-353d2eabd7b8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.441965 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.441937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.441965 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.441958 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.443429 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.443409 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.443983 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.443961 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.450127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.450098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d756e0ed-56b3-473c-8087-353d2eabd7b8-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.450251 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.450186 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg7qq\" (UniqueName: \"kubernetes.io/projected/d756e0ed-56b3-473c-8087-353d2eabd7b8-kube-api-access-mg7qq\") pod \"maas-default-gateway-openshift-default-78bb78ff78-tlnpt\" (UID: \"d756e0ed-56b3-473c-8087-353d2eabd7b8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.525603 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.525513 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:49:59.667865 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.667842 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt"] Jul 24 17:49:59.670352 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:49:59.670324 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd756e0ed_56b3_473c_8087_353d2eabd7b8.slice/crio-87524a1f248c1056315fa6d9f20577621b482a53783111c19694d2c38033d2db WatchSource:0}: Error finding container 87524a1f248c1056315fa6d9f20577621b482a53783111c19694d2c38033d2db: Status 404 returned error can't find the container with id 87524a1f248c1056315fa6d9f20577621b482a53783111c19694d2c38033d2db Jul 24 17:49:59.672326 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.672293 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:49:59.672399 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.672353 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:49:59.672399 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:49:59.672381 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:50:00.561862 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:00.561825 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" event={"ID":"d756e0ed-56b3-473c-8087-353d2eabd7b8","Type":"ContainerStarted","Data":"74d6cc29b6a5496fa4aff8c0a59da73c3e1f5606a01b2bb406f784f5309c7166"} Jul 24 17:50:00.562226 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:00.561882 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" event={"ID":"d756e0ed-56b3-473c-8087-353d2eabd7b8","Type":"ContainerStarted","Data":"87524a1f248c1056315fa6d9f20577621b482a53783111c19694d2c38033d2db"} Jul 24 17:50:00.585424 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:00.585373 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" podStartSLOduration=1.5853581509999999 podStartE2EDuration="1.585358151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:00.583449342 +0000 UTC m=+425.623243383" watchObservedRunningTime="2026-07-24 17:50:00.585358151 +0000 UTC m=+425.625152192" Jul 24 17:50:01.526703 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:01.526665 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:50:01.531588 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:01.531563 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:50:01.566404 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:01.566375 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:50:01.567452 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:01.567436 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:50:13.846574 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.846538 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:50:13.852127 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.852108 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:13.854512 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.854494 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-kfgrn\"" Jul 24 17:50:13.854757 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.854746 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 24 17:50:13.855561 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.855544 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 24 17:50:13.855624 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.855586 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 24 17:50:13.860325 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.860306 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:50:13.892645 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.892612 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:50:13.969108 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.969074 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a98da47f-fd6b-4981-94b4-67d8ceb71d58-config-file\") pod \"limitador-limitador-85d9db4d5d-kmslv\" (UID: \"a98da47f-fd6b-4981-94b4-67d8ceb71d58\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:13.969290 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:13.969166 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bk46\" (UniqueName: \"kubernetes.io/projected/a98da47f-fd6b-4981-94b4-67d8ceb71d58-kube-api-access-8bk46\") pod \"limitador-limitador-85d9db4d5d-kmslv\" (UID: \"a98da47f-fd6b-4981-94b4-67d8ceb71d58\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:14.069622 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:14.069589 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a98da47f-fd6b-4981-94b4-67d8ceb71d58-config-file\") pod \"limitador-limitador-85d9db4d5d-kmslv\" (UID: \"a98da47f-fd6b-4981-94b4-67d8ceb71d58\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:14.069847 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:14.069662 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bk46\" (UniqueName: \"kubernetes.io/projected/a98da47f-fd6b-4981-94b4-67d8ceb71d58-kube-api-access-8bk46\") pod \"limitador-limitador-85d9db4d5d-kmslv\" (UID: \"a98da47f-fd6b-4981-94b4-67d8ceb71d58\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:14.070283 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:14.070259 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/a98da47f-fd6b-4981-94b4-67d8ceb71d58-config-file\") pod \"limitador-limitador-85d9db4d5d-kmslv\" (UID: \"a98da47f-fd6b-4981-94b4-67d8ceb71d58\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:14.078031 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:14.078009 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bk46\" (UniqueName: \"kubernetes.io/projected/a98da47f-fd6b-4981-94b4-67d8ceb71d58-kube-api-access-8bk46\") pod \"limitador-limitador-85d9db4d5d-kmslv\" (UID: \"a98da47f-fd6b-4981-94b4-67d8ceb71d58\") " pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:14.163438 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:14.163343 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:14.299477 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:14.299365 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:50:14.302305 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:50:14.302276 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda98da47f_fd6b_4981_94b4_67d8ceb71d58.slice/crio-c27c7f054aaf40ad5841c8f8d2134152566dc8e94f8e0203971a9a9ec3925fc0 WatchSource:0}: Error finding container c27c7f054aaf40ad5841c8f8d2134152566dc8e94f8e0203971a9a9ec3925fc0: Status 404 returned error can't find the container with id c27c7f054aaf40ad5841c8f8d2134152566dc8e94f8e0203971a9a9ec3925fc0 Jul 24 17:50:14.622692 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:14.622662 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" event={"ID":"a98da47f-fd6b-4981-94b4-67d8ceb71d58","Type":"ContainerStarted","Data":"c27c7f054aaf40ad5841c8f8d2134152566dc8e94f8e0203971a9a9ec3925fc0"} Jul 24 17:50:15.628055 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:15.627962 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" event={"ID":"a98da47f-fd6b-4981-94b4-67d8ceb71d58","Type":"ContainerStarted","Data":"505d327704479e14d5a867acd83507b068e53e29bc5d9629e66fa9afeeab15df"} Jul 24 17:50:15.628055 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:15.628040 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:50:15.650456 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:15.650404 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" podStartSLOduration=1.58341353 podStartE2EDuration="2.650389971s" totalImagesPullingTime="1.066976441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:50:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:50:15.648693346 +0000 UTC m=+440.688487403" watchObservedRunningTime="2026-07-24 17:50:15.650389971 +0000 UTC m=+440.690184059" Jul 24 17:50:26.634584 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:50:26.634514 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-85d9db4d5d-kmslv" Jul 24 17:51:23.197739 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:23.197702 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:51:32.123460 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:32.122849 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tlnpt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": read tcp 10.132.0.2:47696->10.132.0.29:15021: read: connection reset by peer" start-of-body= Jul 24 17:51:32.123460 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:32.122940 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" podUID="d756e0ed-56b3-473c-8087-353d2eabd7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": read tcp 10.132.0.2:47696->10.132.0.29:15021: read: connection reset by peer" Jul 24 17:51:32.975897 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:32.975863 2559 generic.go:356] "Generic (PLEG): container finished" podID="d756e0ed-56b3-473c-8087-353d2eabd7b8" containerID="74d6cc29b6a5496fa4aff8c0a59da73c3e1f5606a01b2bb406f784f5309c7166" exitCode=0 Jul 24 17:51:32.976065 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:32.975940 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" event={"ID":"d756e0ed-56b3-473c-8087-353d2eabd7b8","Type":"ContainerDied","Data":"74d6cc29b6a5496fa4aff8c0a59da73c3e1f5606a01b2bb406f784f5309c7166"} Jul 24 17:51:32.976330 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:32.976314 2559 scope.go:117] "RemoveContainer" containerID="74d6cc29b6a5496fa4aff8c0a59da73c3e1f5606a01b2bb406f784f5309c7166" Jul 24 17:51:32.977283 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:32.977258 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:51:32.977369 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:32.977327 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:51:32.977369 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:32.977358 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 17:51:33.981806 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:33.981750 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" event={"ID":"d756e0ed-56b3-473c-8087-353d2eabd7b8","Type":"ContainerStarted","Data":"dda06891ef2781a8257f2c1011f9bd152d1fe7311e9c6dd3f5daabc1f600bc9c"} Jul 24 17:51:34.526336 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:34.526301 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:51:34.526645 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:34.526622 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tlnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 24 17:51:34.526704 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:34.526674 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" podUID="d756e0ed-56b3-473c-8087-353d2eabd7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 24 17:51:35.526730 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:35.526644 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tlnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 24 17:51:35.527139 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:35.526735 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" podUID="d756e0ed-56b3-473c-8087-353d2eabd7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 24 17:51:36.526025 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:36.525991 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tlnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 24 17:51:36.526194 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:36.526057 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" podUID="d756e0ed-56b3-473c-8087-353d2eabd7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 24 17:51:37.526262 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:37.526229 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tlnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 24 17:51:37.526681 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:37.526283 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" podUID="d756e0ed-56b3-473c-8087-353d2eabd7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 24 17:51:38.526936 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:38.526899 2559 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-tlnpt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" start-of-body= Jul 24 17:51:38.527345 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:38.526953 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" podUID="d756e0ed-56b3-473c-8087-353d2eabd7b8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.29:15021/healthz/ready\": dial tcp 10.132.0.29:15021: connect: connection refused" Jul 24 17:51:39.529634 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:39.529605 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:51:39.530058 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:39.529968 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:51:39.530581 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:51:39.530565 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-tlnpt" Jul 24 17:52:01.480446 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:01.480409 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:52:14.758898 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:14.758866 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:52:21.689015 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:21.688975 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:52:29.792500 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.792467 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw"] Jul 24 17:52:29.796079 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.796058 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:29.806081 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.806058 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-vxck9\"" Jul 24 17:52:29.806206 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.806109 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 24 17:52:29.806206 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.806133 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 24 17:52:29.806206 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.806174 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 24 17:52:29.835431 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.835405 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw"] Jul 24 17:52:29.951727 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.951695 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:29.951946 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.951822 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:29.951946 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.951861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k59zx\" (UniqueName: \"kubernetes.io/projected/cddf6ddc-17d2-461a-8663-e6616fb3c79b-kube-api-access-k59zx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:29.951946 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.951916 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:29.951946 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.951947 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cddf6ddc-17d2-461a-8663-e6616fb3c79b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:29.952136 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:29.951982 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053176 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053082 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053176 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053124 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cddf6ddc-17d2-461a-8663-e6616fb3c79b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053412 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053318 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053475 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053441 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053530 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053511 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053594 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053540 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k59zx\" (UniqueName: \"kubernetes.io/projected/cddf6ddc-17d2-461a-8663-e6616fb3c79b-kube-api-access-k59zx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053683 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053666 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053858 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053835 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.053935 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.053886 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.055662 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.055632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/cddf6ddc-17d2-461a-8663-e6616fb3c79b-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.055823 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.055808 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/cddf6ddc-17d2-461a-8663-e6616fb3c79b-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.071815 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.071767 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k59zx\" (UniqueName: \"kubernetes.io/projected/cddf6ddc-17d2-461a-8663-e6616fb3c79b-kube-api-access-k59zx\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw\" (UID: \"cddf6ddc-17d2-461a-8663-e6616fb3c79b\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.106362 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.106327 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:30.250558 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:30.250530 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw"] Jul 24 17:52:30.252430 ip-10-0-141-191 kubenswrapper[2559]: W0724 17:52:30.252402 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcddf6ddc_17d2_461a_8663_e6616fb3c79b.slice/crio-6ac924f5075a6f537b96b8f56c86cd1a78eb3b3e3b6a703484ccd16c8a99d199 WatchSource:0}: Error finding container 6ac924f5075a6f537b96b8f56c86cd1a78eb3b3e3b6a703484ccd16c8a99d199: Status 404 returned error can't find the container with id 6ac924f5075a6f537b96b8f56c86cd1a78eb3b3e3b6a703484ccd16c8a99d199 Jul 24 17:52:31.091694 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:31.091658 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:52:31.255577 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:31.255553 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" event={"ID":"cddf6ddc-17d2-461a-8663-e6616fb3c79b","Type":"ContainerStarted","Data":"6ac924f5075a6f537b96b8f56c86cd1a78eb3b3e3b6a703484ccd16c8a99d199"} Jul 24 17:52:32.261927 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:32.261890 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" event={"ID":"cddf6ddc-17d2-461a-8663-e6616fb3c79b","Type":"ContainerStarted","Data":"97b8343897d0defef3eaa3796c17f7757fce0a8fcc96bae766f4a9bd58536d51"} Jul 24 17:52:40.106689 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:40.106657 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:40.107317 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:40.106725 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:40.120731 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:40.120702 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:40.161222 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:40.161167 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" podStartSLOduration=10.215325412 podStartE2EDuration="11.161145337s" totalImagesPullingTime="945.819925ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 17:52:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 17:52:32.294412589 +0000 UTC m=+577.334206629" watchObservedRunningTime="2026-07-24 17:52:40.161145337 +0000 UTC m=+585.200939377" Jul 24 17:52:40.306547 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:40.306515 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw" Jul 24 17:52:44.659496 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:44.659452 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:52:55.437397 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:55.437370 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:52:55.439842 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:52:55.439821 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:53:04.965588 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:53:04.965541 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:54:05.059112 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:54:05.059081 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:54:15.392734 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:54:15.392697 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:54:27.079707 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:54:27.079672 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:56:06.156681 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:56:06.156649 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:56:14.565066 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:56:14.565030 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:56:51.249808 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:56:51.249757 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:56:58.259848 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:56:58.259807 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:57:04.655731 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:04.655691 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:57:13.451534 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:13.451498 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:57:22.755662 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:22.755624 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:57:29.452559 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:29.452520 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:57:43.172573 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:43.172539 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:57:48.358537 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:48.358501 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:57:53.450798 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:53.450761 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:57:55.467580 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:55.467552 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:57:55.468059 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:57:55.468041 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 17:58:00.256167 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:58:00.256128 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:58:05.754939 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:58:05.754907 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:58:18.559864 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:58:18.559825 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:58:26.453125 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:58:26.453092 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:58:52.657580 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:58:52.657524 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:59:01.270703 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:59:01.270667 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:59:16.655092 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:59:16.655005 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:59:23.459072 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:59:23.459034 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:59:35.483129 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:59:35.483094 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:59:42.454723 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:59:42.454689 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:59:47.458034 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:59:47.457994 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 17:59:53.163318 ip-10-0-141-191 kubenswrapper[2559]: I0724 17:59:53.163283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:01:41.852989 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:01:41.852954 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:01:48.346612 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:01:48.346578 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:02:05.058459 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:05.058420 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:02:11.853637 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:11.853604 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:02:20.453296 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:20.453205 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:02:25.458302 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:25.458259 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:02:30.756688 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:30.756651 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:02:40.362346 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:40.362311 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:02:45.255164 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:45.255131 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:02:55.500595 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:55.500568 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:02:55.500595 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:55.500581 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:02:55.658180 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:02:55.658145 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:03:00.564034 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:03:00.564000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:03:10.868674 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:03:10.868637 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:03:17.860442 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:03:17.860400 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:03:26.060763 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:03:26.060723 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:03:49.167079 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:03:49.167042 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:03:56.962429 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:03:56.962391 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:04:04.559080 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:04.559042 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:04:06.746908 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.746870 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n"] Jul 24 18:04:06.750327 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.750311 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.753012 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.752993 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-20857293-kserve-self-signed-certs\"" Jul 24 18:04:06.759877 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.759850 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n"] Jul 24 18:04:06.826352 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.826320 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tmp-dir\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.826520 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.826372 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-dshm\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.826520 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.826399 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-model-cache\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.826520 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.826431 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-home\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.826520 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.826473 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tls-certs\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.826520 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.826496 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt56m\" (UniqueName: \"kubernetes.io/projected/68fb77ae-97fa-4378-9b2a-b98da8e9144e-kube-api-access-wt56m\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.927897 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.927866 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tls-certs\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.927897 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.927899 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt56m\" (UniqueName: \"kubernetes.io/projected/68fb77ae-97fa-4378-9b2a-b98da8e9144e-kube-api-access-wt56m\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.928113 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.927932 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tmp-dir\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.928150 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.928112 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-dshm\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.928184 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.928159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-model-cache\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.928228 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.928211 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-home\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.928330 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.928314 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tmp-dir\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.928453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.928427 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-model-cache\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.928518 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.928504 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-home\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.930486 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.930463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-dshm\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.930643 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.930626 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tls-certs\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:06.936071 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:06.936050 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt56m\" (UniqueName: \"kubernetes.io/projected/68fb77ae-97fa-4378-9b2a-b98da8e9144e-kube-api-access-wt56m\") pod \"e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:07.060183 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:07.060087 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:07.209090 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:07.209067 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n"] Jul 24 18:04:07.211939 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:04:07.211910 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod68fb77ae_97fa_4378_9b2a_b98da8e9144e.slice/crio-e9c4ea5e375e0d9b5cba2a487c459f734c0d15d84c4b944bfb5503e73dc79986 WatchSource:0}: Error finding container e9c4ea5e375e0d9b5cba2a487c459f734c0d15d84c4b944bfb5503e73dc79986: Status 404 returned error can't find the container with id e9c4ea5e375e0d9b5cba2a487c459f734c0d15d84c4b944bfb5503e73dc79986 Jul 24 18:04:07.347146 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:07.347114 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" event={"ID":"68fb77ae-97fa-4378-9b2a-b98da8e9144e","Type":"ContainerStarted","Data":"d320e40d438bcfcd6d136baa4f81b04fcac40d2b56d0d0cf418afc1cc3a9a312"} Jul 24 18:04:07.347315 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:07.347156 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" event={"ID":"68fb77ae-97fa-4378-9b2a-b98da8e9144e","Type":"ContainerStarted","Data":"e9c4ea5e375e0d9b5cba2a487c459f734c0d15d84c4b944bfb5503e73dc79986"} Jul 24 18:04:07.376230 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:07.376178 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" podStartSLOduration=1.376158357 podStartE2EDuration="1.376158357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:04:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:04:07.374993097 +0000 UTC m=+1272.414787138" watchObservedRunningTime="2026-07-24 18:04:07.376158357 +0000 UTC m=+1272.415952410" Jul 24 18:04:15.664425 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:15.664387 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:04:17.060354 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:17.060313 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:17.060883 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:17.060382 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:17.074643 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:17.074616 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:17.401817 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:17.401770 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:31.391597 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:31.391565 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n"] Jul 24 18:04:33.556319 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:33.556284 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:04:43.457974 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:43.457938 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:04:46.411017 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.410942 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" podUID="68fb77ae-97fa-4378-9b2a-b98da8e9144e" containerName="main" containerID="cri-o://d320e40d438bcfcd6d136baa4f81b04fcac40d2b56d0d0cf418afc1cc3a9a312" gracePeriod=45 Jul 24 18:04:46.528424 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.528394 2559 generic.go:356] "Generic (PLEG): container finished" podID="68fb77ae-97fa-4378-9b2a-b98da8e9144e" containerID="d320e40d438bcfcd6d136baa4f81b04fcac40d2b56d0d0cf418afc1cc3a9a312" exitCode=0 Jul 24 18:04:46.528569 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.528444 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" event={"ID":"68fb77ae-97fa-4378-9b2a-b98da8e9144e","Type":"ContainerDied","Data":"d320e40d438bcfcd6d136baa4f81b04fcac40d2b56d0d0cf418afc1cc3a9a312"} Jul 24 18:04:46.665719 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.665658 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:46.775241 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775201 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-home\") pod \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " Jul 24 18:04:46.775408 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775270 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tls-certs\") pod \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " Jul 24 18:04:46.775408 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775335 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tmp-dir\") pod \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " Jul 24 18:04:46.775408 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775376 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-model-cache\") pod \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " Jul 24 18:04:46.775537 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775408 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt56m\" (UniqueName: \"kubernetes.io/projected/68fb77ae-97fa-4378-9b2a-b98da8e9144e-kube-api-access-wt56m\") pod \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " Jul 24 18:04:46.775537 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775442 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-dshm\") pod \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\" (UID: \"68fb77ae-97fa-4378-9b2a-b98da8e9144e\") " Jul 24 18:04:46.775636 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775550 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-home" (OuterVolumeSpecName: "home") pod "68fb77ae-97fa-4378-9b2a-b98da8e9144e" (UID: "68fb77ae-97fa-4378-9b2a-b98da8e9144e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:04:46.775636 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775577 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "68fb77ae-97fa-4378-9b2a-b98da8e9144e" (UID: "68fb77ae-97fa-4378-9b2a-b98da8e9144e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:04:46.775636 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775626 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-model-cache" (OuterVolumeSpecName: "model-cache") pod "68fb77ae-97fa-4378-9b2a-b98da8e9144e" (UID: "68fb77ae-97fa-4378-9b2a-b98da8e9144e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:04:46.775808 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775743 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:04:46.775808 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775757 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:04:46.775808 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.775767 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:04:46.777705 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.777676 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-dshm" (OuterVolumeSpecName: "dshm") pod "68fb77ae-97fa-4378-9b2a-b98da8e9144e" (UID: "68fb77ae-97fa-4378-9b2a-b98da8e9144e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:04:46.777705 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.777677 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "68fb77ae-97fa-4378-9b2a-b98da8e9144e" (UID: "68fb77ae-97fa-4378-9b2a-b98da8e9144e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:04:46.777882 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.777758 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68fb77ae-97fa-4378-9b2a-b98da8e9144e-kube-api-access-wt56m" (OuterVolumeSpecName: "kube-api-access-wt56m") pod "68fb77ae-97fa-4378-9b2a-b98da8e9144e" (UID: "68fb77ae-97fa-4378-9b2a-b98da8e9144e"). InnerVolumeSpecName "kube-api-access-wt56m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:04:46.876528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.876486 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt56m\" (UniqueName: \"kubernetes.io/projected/68fb77ae-97fa-4378-9b2a-b98da8e9144e-kube-api-access-wt56m\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:04:46.876528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.876520 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/68fb77ae-97fa-4378-9b2a-b98da8e9144e-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:04:46.876528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:46.876530 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/68fb77ae-97fa-4378-9b2a-b98da8e9144e-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:04:47.534310 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:47.534283 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" Jul 24 18:04:47.534756 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:47.534286 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n" event={"ID":"68fb77ae-97fa-4378-9b2a-b98da8e9144e","Type":"ContainerDied","Data":"e9c4ea5e375e0d9b5cba2a487c459f734c0d15d84c4b944bfb5503e73dc79986"} Jul 24 18:04:47.534756 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:47.534407 2559 scope.go:117] "RemoveContainer" containerID="d320e40d438bcfcd6d136baa4f81b04fcac40d2b56d0d0cf418afc1cc3a9a312" Jul 24 18:04:47.551390 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:47.551366 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n"] Jul 24 18:04:47.561232 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:47.561206 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-20857293-kserve-7586fc8fb7-wqv2n"] Jul 24 18:04:49.355822 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:49.355774 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:04:49.494242 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:49.494202 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="68fb77ae-97fa-4378-9b2a-b98da8e9144e" path="/var/lib/kubelet/pods/68fb77ae-97fa-4378-9b2a-b98da8e9144e/volumes" Jul 24 18:04:55.918943 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:04:55.918909 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:05:04.663149 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:05:04.663116 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:05:15.953467 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:05:15.953429 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:05:24.156873 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:05:24.156752 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:05:36.563083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:05:36.563043 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:05:42.656260 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:05:42.656224 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:05:50.965918 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:05:50.965884 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:05:56.554644 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:05:56.554609 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:06:05.256853 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:06:05.256821 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:06:13.649926 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:06:13.649887 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:06:24.854652 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:06:24.854617 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:06:34.150028 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:06:34.149981 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:06:44.353215 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:06:44.353181 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:06:52.457690 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:06:52.457647 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:07:00.073389 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:00.073352 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:07:08.767286 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:08.767248 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:07:22.656797 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:22.656760 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:07:31.453301 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:31.453254 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:07:37.957250 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:37.957212 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:07:48.159685 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:48.159645 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:07:53.160665 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:53.160624 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:07:55.527974 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:55.527946 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:07:55.529190 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:07:55.529167 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:08:00.570509 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:00.570475 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:08:11.853756 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:11.853720 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:08:34.124434 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.124352 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f"] Jul 24 18:08:34.124907 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.124676 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68fb77ae-97fa-4378-9b2a-b98da8e9144e" containerName="main" Jul 24 18:08:34.124907 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.124686 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="68fb77ae-97fa-4378-9b2a-b98da8e9144e" containerName="main" Jul 24 18:08:34.124907 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.124750 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="68fb77ae-97fa-4378-9b2a-b98da8e9144e" containerName="main" Jul 24 18:08:34.127519 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.127502 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" Jul 24 18:08:34.130840 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.130816 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:08:34.131502 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.131484 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:08:34.140064 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.140038 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f"] Jul 24 18:08:34.176559 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.176528 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34402ebb-aea4-4d61-a68c-5a89b50af8eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-096012db-jxt9f\" (UID: \"34402ebb-aea4-4d61-a68c-5a89b50af8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" Jul 24 18:08:34.278039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.278001 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34402ebb-aea4-4d61-a68c-5a89b50af8eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-096012db-jxt9f\" (UID: \"34402ebb-aea4-4d61-a68c-5a89b50af8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" Jul 24 18:08:34.278636 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.278617 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34402ebb-aea4-4d61-a68c-5a89b50af8eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-096012db-jxt9f\" (UID: \"34402ebb-aea4-4d61-a68c-5a89b50af8eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" Jul 24 18:08:34.436989 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.436891 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" Jul 24 18:08:34.570990 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.570962 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f"] Jul 24 18:08:34.572858 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:08:34.572832 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34402ebb_aea4_4d61_a68c_5a89b50af8eb.slice/crio-cda1cbec445222a05fcfa3255758256fd1b9782a90ae96c05d8c68eca8cb6361 WatchSource:0}: Error finding container cda1cbec445222a05fcfa3255758256fd1b9782a90ae96c05d8c68eca8cb6361: Status 404 returned error can't find the container with id cda1cbec445222a05fcfa3255758256fd1b9782a90ae96c05d8c68eca8cb6361 Jul 24 18:08:34.574802 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:34.574772 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 18:08:35.562990 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:35.562953 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" event={"ID":"34402ebb-aea4-4d61-a68c-5a89b50af8eb","Type":"ContainerStarted","Data":"cda1cbec445222a05fcfa3255758256fd1b9782a90ae96c05d8c68eca8cb6361"} Jul 24 18:08:37.574718 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:37.574689 2559 generic.go:356] "Generic (PLEG): container finished" podID="34402ebb-aea4-4d61-a68c-5a89b50af8eb" containerID="4a1e702752a2cfbd640696e535735b4dbf5bb6ee8d394f63e8c2c096381e0191" exitCode=0 Jul 24 18:08:37.575100 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:37.574765 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" event={"ID":"34402ebb-aea4-4d61-a68c-5a89b50af8eb","Type":"ContainerDied","Data":"4a1e702752a2cfbd640696e535735b4dbf5bb6ee8d394f63e8c2c096381e0191"} Jul 24 18:08:38.714357 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:38.714335 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" Jul 24 18:08:38.814043 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:38.814008 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34402ebb-aea4-4d61-a68c-5a89b50af8eb-maas-api-service-ca\") pod \"34402ebb-aea4-4d61-a68c-5a89b50af8eb\" (UID: \"34402ebb-aea4-4d61-a68c-5a89b50af8eb\") " Jul 24 18:08:38.814385 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:38.814364 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34402ebb-aea4-4d61-a68c-5a89b50af8eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34402ebb-aea4-4d61-a68c-5a89b50af8eb" (UID: "34402ebb-aea4-4d61-a68c-5a89b50af8eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:08:38.914969 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:38.914886 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34402ebb-aea4-4d61-a68c-5a89b50af8eb-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:08:39.585453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:39.585421 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" Jul 24 18:08:39.585453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:39.585424 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f" event={"ID":"34402ebb-aea4-4d61-a68c-5a89b50af8eb","Type":"ContainerDied","Data":"cda1cbec445222a05fcfa3255758256fd1b9782a90ae96c05d8c68eca8cb6361"} Jul 24 18:08:39.585669 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:39.585475 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cda1cbec445222a05fcfa3255758256fd1b9782a90ae96c05d8c68eca8cb6361" Jul 24 18:08:47.058841 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:47.058804 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:08:49.217469 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:49.217429 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f"] Jul 24 18:08:49.220388 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:49.220360 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-096012db-jxt9f"] Jul 24 18:08:49.493684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:49.493605 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34402ebb-aea4-4d61-a68c-5a89b50af8eb" path="/var/lib/kubelet/pods/34402ebb-aea4-4d61-a68c-5a89b50af8eb/volumes" Jul 24 18:08:56.563652 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:08:56.563607 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:09:02.505955 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.505916 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw"] Jul 24 18:09:02.506536 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.506432 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34402ebb-aea4-4d61-a68c-5a89b50af8eb" containerName="revoke-keys" Jul 24 18:09:02.506536 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.506455 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="34402ebb-aea4-4d61-a68c-5a89b50af8eb" containerName="revoke-keys" Jul 24 18:09:02.506660 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.506561 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="34402ebb-aea4-4d61-a68c-5a89b50af8eb" containerName="revoke-keys" Jul 24 18:09:02.511089 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.511067 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" Jul 24 18:09:02.514297 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.514269 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:09:02.514427 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.514409 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:09:02.518891 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.518546 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw"] Jul 24 18:09:02.625043 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.625012 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ee56f3-c619-4712-8f4e-f269ef0cf3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw\" (UID: \"92ee56f3-c619-4712-8f4e-f269ef0cf3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" Jul 24 18:09:02.725928 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.725891 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ee56f3-c619-4712-8f4e-f269ef0cf3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw\" (UID: \"92ee56f3-c619-4712-8f4e-f269ef0cf3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" Jul 24 18:09:02.726493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.726474 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ee56f3-c619-4712-8f4e-f269ef0cf3d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw\" (UID: \"92ee56f3-c619-4712-8f4e-f269ef0cf3d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" Jul 24 18:09:02.822537 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.822512 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" Jul 24 18:09:02.959135 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:02.959111 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw"] Jul 24 18:09:02.961022 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:09:02.960999 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ee56f3_c619_4712_8f4e_f269ef0cf3d1.slice/crio-dfdb4ee6a506e5b4b6ddc329e1fb8c39a5097ee7584328d7c95701649d264b4d WatchSource:0}: Error finding container dfdb4ee6a506e5b4b6ddc329e1fb8c39a5097ee7584328d7c95701649d264b4d: Status 404 returned error can't find the container with id dfdb4ee6a506e5b4b6ddc329e1fb8c39a5097ee7584328d7c95701649d264b4d Jul 24 18:09:03.691689 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:03.691658 2559 generic.go:356] "Generic (PLEG): container finished" podID="92ee56f3-c619-4712-8f4e-f269ef0cf3d1" containerID="a6dd6373eceb6ff3f6f4fe9148664a6e59ece93098b69037bf6df2cb6707c1d5" exitCode=0 Jul 24 18:09:03.692130 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:03.691751 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" event={"ID":"92ee56f3-c619-4712-8f4e-f269ef0cf3d1","Type":"ContainerDied","Data":"a6dd6373eceb6ff3f6f4fe9148664a6e59ece93098b69037bf6df2cb6707c1d5"} Jul 24 18:09:03.692130 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:03.691827 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" event={"ID":"92ee56f3-c619-4712-8f4e-f269ef0cf3d1","Type":"ContainerStarted","Data":"dfdb4ee6a506e5b4b6ddc329e1fb8c39a5097ee7584328d7c95701649d264b4d"} Jul 24 18:09:04.836262 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:04.836236 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" Jul 24 18:09:04.945468 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:04.945424 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ee56f3-c619-4712-8f4e-f269ef0cf3d1-maas-api-service-ca\") pod \"92ee56f3-c619-4712-8f4e-f269ef0cf3d1\" (UID: \"92ee56f3-c619-4712-8f4e-f269ef0cf3d1\") " Jul 24 18:09:04.945841 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:04.945818 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ee56f3-c619-4712-8f4e-f269ef0cf3d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92ee56f3-c619-4712-8f4e-f269ef0cf3d1" (UID: "92ee56f3-c619-4712-8f4e-f269ef0cf3d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:09:05.046837 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:05.046732 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92ee56f3-c619-4712-8f4e-f269ef0cf3d1-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:09:05.702678 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:05.702650 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" Jul 24 18:09:05.702678 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:05.702670 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw" event={"ID":"92ee56f3-c619-4712-8f4e-f269ef0cf3d1","Type":"ContainerDied","Data":"dfdb4ee6a506e5b4b6ddc329e1fb8c39a5097ee7584328d7c95701649d264b4d"} Jul 24 18:09:05.702913 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:05.702710 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dfdb4ee6a506e5b4b6ddc329e1fb8c39a5097ee7584328d7c95701649d264b4d" Jul 24 18:09:15.249225 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:15.249190 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:09:17.728321 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:17.728280 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw"] Jul 24 18:09:17.734393 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:17.734367 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-35d23e85-8t6fw"] Jul 24 18:09:18.635755 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.635724 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r"] Jul 24 18:09:18.636097 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.636083 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92ee56f3-c619-4712-8f4e-f269ef0cf3d1" containerName="revoke-keys" Jul 24 18:09:18.636144 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.636099 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="92ee56f3-c619-4712-8f4e-f269ef0cf3d1" containerName="revoke-keys" Jul 24 18:09:18.636178 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.636150 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="92ee56f3-c619-4712-8f4e-f269ef0cf3d1" containerName="revoke-keys" Jul 24 18:09:18.639281 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.639265 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.641916 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.641898 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-f37130a8-openshift-default-dockercfg-2v4fb\"" Jul 24 18:09:18.651491 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.651467 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r"] Jul 24 18:09:18.766551 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766517 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-podinfo\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.766551 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-certs\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.767122 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-data\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.767122 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766626 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istiod-ca-cert\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.767122 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766671 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-token\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.767122 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766709 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rv4x\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-kube-api-access-6rv4x\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.767122 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-socket\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.767122 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766758 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-envoy\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.767122 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.766832 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-credential-socket\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868216 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868183 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-data\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868216 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868217 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istiod-ca-cert\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868471 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868238 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-token\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868471 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rv4x\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-kube-api-access-6rv4x\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868471 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868380 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-socket\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868471 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868413 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-envoy\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868471 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868459 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-credential-socket\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868735 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868562 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-podinfo\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.868735 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-certs\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.869020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.868988 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-data\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.869020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.869003 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-credential-socket\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.869020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.869015 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-certs\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.869230 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.869114 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-socket\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.869510 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.869491 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istiod-ca-cert\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.870822 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.870804 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-envoy\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.871169 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.871151 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-podinfo\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.876729 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.876709 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rv4x\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-kube-api-access-6rv4x\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.876894 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.876873 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-token\") pod \"e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:18.949702 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:18.949620 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:19.090727 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.090704 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r"] Jul 24 18:09:19.093149 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:09:19.093118 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0db9a6c_b8e9_40b0_85a9_e0b5cc6fce14.slice/crio-8f46d8d61f83c758be57408057665078640902deef0903c28bae93bbd0fe47ac WatchSource:0}: Error finding container 8f46d8d61f83c758be57408057665078640902deef0903c28bae93bbd0fe47ac: Status 404 returned error can't find the container with id 8f46d8d61f83c758be57408057665078640902deef0903c28bae93bbd0fe47ac Jul 24 18:09:19.095358 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.095321 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:09:19.095461 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.095388 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:09:19.095461 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.095414 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:09:19.495472 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.495430 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92ee56f3-c619-4712-8f4e-f269ef0cf3d1" path="/var/lib/kubelet/pods/92ee56f3-c619-4712-8f4e-f269ef0cf3d1/volumes" Jul 24 18:09:19.766573 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.766476 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" event={"ID":"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14","Type":"ContainerStarted","Data":"1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b"} Jul 24 18:09:19.766573 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.766532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" event={"ID":"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14","Type":"ContainerStarted","Data":"8f46d8d61f83c758be57408057665078640902deef0903c28bae93bbd0fe47ac"} Jul 24 18:09:19.809344 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.809288 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" podStartSLOduration=1.809271721 podStartE2EDuration="1.809271721s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:09:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:09:19.808506777 +0000 UTC m=+1584.848300817" watchObservedRunningTime="2026-07-24 18:09:19.809271721 +0000 UTC m=+1584.849065762" Jul 24 18:09:19.950301 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:19.950270 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:20.955205 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:20.955177 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:21.774773 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:21.774746 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:21.775808 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:21.775775 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:24.849608 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:24.849566 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:09:34.853367 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:34.853337 2559 generic.go:356] "Generic (PLEG): container finished" podID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerID="1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b" exitCode=0 Jul 24 18:09:34.853874 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:34.853389 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" event={"ID":"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14","Type":"ContainerDied","Data":"1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b"} Jul 24 18:09:34.853874 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:34.853768 2559 scope.go:117] "RemoveContainer" containerID="1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b" Jul 24 18:09:34.854746 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:34.854720 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:09:34.854817 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:34.854803 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:09:34.854861 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:34.854838 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:09:35.859545 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:35.859513 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" event={"ID":"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14","Type":"ContainerStarted","Data":"1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6"} Jul 24 18:09:35.950436 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:35.950399 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:35.951473 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:35.951446 2559 patch_prober.go:28] interesting pod/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 24 18:09:35.951593 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:35.951512 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 24 18:09:36.775520 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:36.775470 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:36.950942 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:36.950900 2559 patch_prober.go:28] interesting pod/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 24 18:09:36.951409 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:36.950976 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 24 18:09:37.951075 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:37.951042 2559 patch_prober.go:28] interesting pod/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 24 18:09:37.951468 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:37.951094 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 24 18:09:38.950289 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:38.950254 2559 patch_prober.go:28] interesting pod/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 24 18:09:38.950495 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:38.950318 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 24 18:09:39.950589 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:39.950554 2559 patch_prober.go:28] interesting pod/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" start-of-body= Jul 24 18:09:39.951020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:39.950612 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.35:15021/healthz/ready\": dial tcp 10.132.0.35:15021: connect: connection refused" Jul 24 18:09:40.954210 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:40.954183 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:40.955057 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:40.955036 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:09:59.655526 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:09:59.655492 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:10:03.453077 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:03.453043 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r"] Jul 24 18:10:03.453462 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:03.453250 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" containerID="cri-o://1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6" gracePeriod=30 Jul 24 18:10:08.713945 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.713882 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:10:08.811515 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811488 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-data\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811709 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811549 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-podinfo\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811709 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811601 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-envoy\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811709 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811644 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-certs\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811709 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811666 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-socket\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811715 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-credential-socket\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811749 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-token\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811766 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-data" (OuterVolumeSpecName: "istio-data") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:08.811954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811816 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istiod-ca-cert\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811841 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6rv4x\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-kube-api-access-6rv4x\") pod \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\" (UID: \"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14\") " Jul 24 18:10:08.811954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811937 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:08.812242 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811973 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:08.812242 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.811984 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:08.812242 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.812213 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-data\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:08.812242 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.812233 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:08.812242 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.812232 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:10:08.812653 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.812248 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-workload-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:08.812653 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.812261 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-credential-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:08.814249 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.814223 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:10:08.814249 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.814229 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-kube-api-access-6rv4x" (OuterVolumeSpecName: "kube-api-access-6rv4x") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "kube-api-access-6rv4x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:10:08.814421 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.814251 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:10:08.814421 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.814287 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" (UID: "d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:10:08.913214 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.913166 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-podinfo\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:08.913214 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.913209 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-envoy\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:08.913214 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.913218 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istio-token\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:08.913214 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.913227 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-istiod-ca-cert\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:08.913214 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:08.913236 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rv4x\" (UniqueName: \"kubernetes.io/projected/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14-kube-api-access-6rv4x\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:09.012607 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.012514 2559 generic.go:356] "Generic (PLEG): container finished" podID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerID="1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6" exitCode=0 Jul 24 18:10:09.012607 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.012576 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" Jul 24 18:10:09.012607 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.012595 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" event={"ID":"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14","Type":"ContainerDied","Data":"1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6"} Jul 24 18:10:09.012871 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.012650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r" event={"ID":"d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14","Type":"ContainerDied","Data":"8f46d8d61f83c758be57408057665078640902deef0903c28bae93bbd0fe47ac"} Jul 24 18:10:09.012871 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.012672 2559 scope.go:117] "RemoveContainer" containerID="1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6" Jul 24 18:10:09.022268 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.022249 2559 scope.go:117] "RemoveContainer" containerID="1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b" Jul 24 18:10:09.030270 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.030252 2559 scope.go:117] "RemoveContainer" containerID="1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6" Jul 24 18:10:09.030522 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:10:09.030501 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6\": container with ID starting with 1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6 not found: ID does not exist" containerID="1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6" Jul 24 18:10:09.030570 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.030530 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6"} err="failed to get container status \"1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6\": rpc error: code = NotFound desc = could not find container \"1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6\": container with ID starting with 1814a76ae96c89d79570e14196a3dc27879fd99e8d42bb136661ead5acb0deb6 not found: ID does not exist" Jul 24 18:10:09.030570 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.030548 2559 scope.go:117] "RemoveContainer" containerID="1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b" Jul 24 18:10:09.030758 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:10:09.030741 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b\": container with ID starting with 1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b not found: ID does not exist" containerID="1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b" Jul 24 18:10:09.030836 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.030768 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b"} err="failed to get container status \"1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b\": rpc error: code = NotFound desc = could not find container \"1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b\": container with ID starting with 1f1c576af73e862574a9d9b36f0286a7e2592d76881d9d0898cd8c2b511c0c3b not found: ID does not exist" Jul 24 18:10:09.038103 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.038078 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r"] Jul 24 18:10:09.046092 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.046069 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-f37130a8-openshift-default-54b9444f74-pk55r"] Jul 24 18:10:09.493340 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:09.493311 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" path="/var/lib/kubelet/pods/d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14/volumes" Jul 24 18:10:11.217716 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:11.217679 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:10:32.072738 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.072707 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q"] Jul 24 18:10:32.073128 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.073069 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" Jul 24 18:10:32.073128 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.073086 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" Jul 24 18:10:32.073128 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.073104 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" Jul 24 18:10:32.073128 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.073110 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" Jul 24 18:10:32.073254 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.073171 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" Jul 24 18:10:32.076074 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.076057 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" Jul 24 18:10:32.078975 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.078953 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:10:32.078975 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.078972 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:10:32.095980 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.095938 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q"] Jul 24 18:10:32.122645 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.122619 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b7909-1358-4376-a6e6-77649cf282a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-294b6e13-rls5q\" (UID: \"932b7909-1358-4376-a6e6-77649cf282a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" Jul 24 18:10:32.223061 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.223018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b7909-1358-4376-a6e6-77649cf282a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-294b6e13-rls5q\" (UID: \"932b7909-1358-4376-a6e6-77649cf282a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" Jul 24 18:10:32.223670 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.223651 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b7909-1358-4376-a6e6-77649cf282a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-294b6e13-rls5q\" (UID: \"932b7909-1358-4376-a6e6-77649cf282a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" Jul 24 18:10:32.386334 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.386254 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" Jul 24 18:10:32.529388 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:32.529361 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q"] Jul 24 18:10:32.531974 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:10:32.531946 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod932b7909_1358_4376_a6e6_77649cf282a4.slice/crio-f68f6c02e22b5298bfd5ff59056f2001df4f07507c5acdfe2d00ff6bff432042 WatchSource:0}: Error finding container f68f6c02e22b5298bfd5ff59056f2001df4f07507c5acdfe2d00ff6bff432042: Status 404 returned error can't find the container with id f68f6c02e22b5298bfd5ff59056f2001df4f07507c5acdfe2d00ff6bff432042 Jul 24 18:10:33.125713 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:33.125682 2559 generic.go:356] "Generic (PLEG): container finished" podID="932b7909-1358-4376-a6e6-77649cf282a4" containerID="9deb00a47ccaf9dedfc39e9586f7776938ef33ae4e54474d6fea7172fea6557f" exitCode=0 Jul 24 18:10:33.126061 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:33.125760 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" event={"ID":"932b7909-1358-4376-a6e6-77649cf282a4","Type":"ContainerDied","Data":"9deb00a47ccaf9dedfc39e9586f7776938ef33ae4e54474d6fea7172fea6557f"} Jul 24 18:10:33.126061 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:33.125880 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" event={"ID":"932b7909-1358-4376-a6e6-77649cf282a4","Type":"ContainerStarted","Data":"f68f6c02e22b5298bfd5ff59056f2001df4f07507c5acdfe2d00ff6bff432042"} Jul 24 18:10:34.274668 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:34.274643 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" Jul 24 18:10:34.343572 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:34.343537 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b7909-1358-4376-a6e6-77649cf282a4-maas-api-service-ca\") pod \"932b7909-1358-4376-a6e6-77649cf282a4\" (UID: \"932b7909-1358-4376-a6e6-77649cf282a4\") " Jul 24 18:10:34.343914 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:34.343892 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/932b7909-1358-4376-a6e6-77649cf282a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "932b7909-1358-4376-a6e6-77649cf282a4" (UID: "932b7909-1358-4376-a6e6-77649cf282a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:10:34.444687 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:34.444606 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/932b7909-1358-4376-a6e6-77649cf282a4-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:10:35.135984 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:35.135952 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" event={"ID":"932b7909-1358-4376-a6e6-77649cf282a4","Type":"ContainerDied","Data":"f68f6c02e22b5298bfd5ff59056f2001df4f07507c5acdfe2d00ff6bff432042"} Jul 24 18:10:35.135984 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:35.135985 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q" Jul 24 18:10:35.136194 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:35.135999 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f68f6c02e22b5298bfd5ff59056f2001df4f07507c5acdfe2d00ff6bff432042" Jul 24 18:10:44.752981 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:44.752945 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:10:47.154957 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:47.154920 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q"] Jul 24 18:10:47.158418 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:47.158394 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-294b6e13-rls5q"] Jul 24 18:10:47.493848 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:47.493733 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="932b7909-1358-4376-a6e6-77649cf282a4" path="/var/lib/kubelet/pods/932b7909-1358-4376-a6e6-77649cf282a4/volumes" Jul 24 18:10:50.559884 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:50.559853 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:10:58.559909 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:10:58.559872 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:12:12.289972 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:12.289934 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:12:20.174638 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:20.174605 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:12:29.754951 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:29.754911 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:12:38.655916 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:38.655879 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:12:42.510014 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.509982 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m"] Jul 24 18:12:42.510419 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.510311 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="932b7909-1358-4376-a6e6-77649cf282a4" containerName="revoke-keys" Jul 24 18:12:42.510419 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.510321 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="932b7909-1358-4376-a6e6-77649cf282a4" containerName="revoke-keys" Jul 24 18:12:42.510419 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.510388 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0db9a6c-b8e9-40b0-85a9-e0b5cc6fce14" containerName="istio-proxy" Jul 24 18:12:42.510419 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.510399 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="932b7909-1358-4376-a6e6-77649cf282a4" containerName="revoke-keys" Jul 24 18:12:42.513515 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.513499 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" Jul 24 18:12:42.516275 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.516254 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:12:42.516438 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.516417 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:12:42.522978 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.522949 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m"] Jul 24 18:12:42.589948 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.589919 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8656e9-22fe-4ee6-a344-ec1cfbb6c982-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m\" (UID: \"1b8656e9-22fe-4ee6-a344-ec1cfbb6c982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" Jul 24 18:12:42.691271 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.691231 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8656e9-22fe-4ee6-a344-ec1cfbb6c982-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m\" (UID: \"1b8656e9-22fe-4ee6-a344-ec1cfbb6c982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" Jul 24 18:12:42.691885 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.691866 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8656e9-22fe-4ee6-a344-ec1cfbb6c982-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m\" (UID: \"1b8656e9-22fe-4ee6-a344-ec1cfbb6c982\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" Jul 24 18:12:42.824588 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.824563 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" Jul 24 18:12:42.956105 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:42.956080 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m"] Jul 24 18:12:42.958627 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:12:42.958601 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b8656e9_22fe_4ee6_a344_ec1cfbb6c982.slice/crio-0f3c3977065c10b00859d4698c7c35d5dc854a182ddad335641e1c3fe49691a8 WatchSource:0}: Error finding container 0f3c3977065c10b00859d4698c7c35d5dc854a182ddad335641e1c3fe49691a8: Status 404 returned error can't find the container with id 0f3c3977065c10b00859d4698c7c35d5dc854a182ddad335641e1c3fe49691a8 Jul 24 18:12:43.732228 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:43.732189 2559 generic.go:356] "Generic (PLEG): container finished" podID="1b8656e9-22fe-4ee6-a344-ec1cfbb6c982" containerID="e1f9b15c660dddc2f47a3e7d29cb4b8b04964df9b32790fed37f040cfe8959ba" exitCode=0 Jul 24 18:12:43.732583 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:43.732269 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" event={"ID":"1b8656e9-22fe-4ee6-a344-ec1cfbb6c982","Type":"ContainerDied","Data":"e1f9b15c660dddc2f47a3e7d29cb4b8b04964df9b32790fed37f040cfe8959ba"} Jul 24 18:12:43.732583 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:43.732328 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" event={"ID":"1b8656e9-22fe-4ee6-a344-ec1cfbb6c982","Type":"ContainerStarted","Data":"0f3c3977065c10b00859d4698c7c35d5dc854a182ddad335641e1c3fe49691a8"} Jul 24 18:12:44.871110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:44.871087 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" Jul 24 18:12:44.911438 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:44.911410 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8656e9-22fe-4ee6-a344-ec1cfbb6c982-maas-api-service-ca\") pod \"1b8656e9-22fe-4ee6-a344-ec1cfbb6c982\" (UID: \"1b8656e9-22fe-4ee6-a344-ec1cfbb6c982\") " Jul 24 18:12:44.911757 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:44.911734 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b8656e9-22fe-4ee6-a344-ec1cfbb6c982-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b8656e9-22fe-4ee6-a344-ec1cfbb6c982" (UID: "1b8656e9-22fe-4ee6-a344-ec1cfbb6c982"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:12:45.012622 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:45.012544 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b8656e9-22fe-4ee6-a344-ec1cfbb6c982-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:12:45.743186 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:45.743157 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" Jul 24 18:12:45.743186 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:45.743169 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m" event={"ID":"1b8656e9-22fe-4ee6-a344-ec1cfbb6c982","Type":"ContainerDied","Data":"0f3c3977065c10b00859d4698c7c35d5dc854a182ddad335641e1c3fe49691a8"} Jul 24 18:12:45.743399 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:45.743216 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f3c3977065c10b00859d4698c7c35d5dc854a182ddad335641e1c3fe49691a8" Jul 24 18:12:55.063546 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:55.063504 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:12:55.556679 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:55.556598 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:12:55.557364 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:55.557343 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:12:57.597701 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:57.597656 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m"] Jul 24 18:12:57.600723 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:57.600699 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-72f7ddbd-z6c8m"] Jul 24 18:12:59.494170 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:12:59.494135 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8656e9-22fe-4ee6-a344-ec1cfbb6c982" path="/var/lib/kubelet/pods/1b8656e9-22fe-4ee6-a344-ec1cfbb6c982/volumes" Jul 24 18:13:32.154458 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:32.154420 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:13:42.142051 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.142013 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9"] Jul 24 18:13:42.142619 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.142340 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8656e9-22fe-4ee6-a344-ec1cfbb6c982" containerName="revoke-keys" Jul 24 18:13:42.142619 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.142351 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8656e9-22fe-4ee6-a344-ec1cfbb6c982" containerName="revoke-keys" Jul 24 18:13:42.142619 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.142410 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8656e9-22fe-4ee6-a344-ec1cfbb6c982" containerName="revoke-keys" Jul 24 18:13:42.145462 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.145444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.148243 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.148217 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-159379b5\"/\"openshift-service-ca.crt\"" Jul 24 18:13:42.148409 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.148389 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-159379b5\"/\"e2e-lifecycle-model-159379b5-kserve-self-signed-certs\"" Jul 24 18:13:42.148724 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.148701 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-159379b5\"/\"kube-root-ca.crt\"" Jul 24 18:13:42.148854 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.148803 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-159379b5\"/\"default-dockercfg-prtpw\"" Jul 24 18:13:42.155884 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.155861 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9"] Jul 24 18:13:42.201693 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.201664 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtngq\" (UniqueName: \"kubernetes.io/projected/2f38e7a3-0a32-41f6-b51f-df2fcc370421-kube-api-access-vtngq\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.201889 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.201757 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-home\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.201889 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.201810 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-dshm\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.201996 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.201880 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tmp-dir\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.201996 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.201925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tls-certs\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.201996 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.201952 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-model-cache\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.302639 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.302605 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tmp-dir\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.302639 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.302642 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tls-certs\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.302929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.302663 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-model-cache\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.302929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.302696 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtngq\" (UniqueName: \"kubernetes.io/projected/2f38e7a3-0a32-41f6-b51f-df2fcc370421-kube-api-access-vtngq\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.302929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.302781 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-home\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.302929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.302834 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-dshm\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.303141 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.303095 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tmp-dir\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.303182 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.303137 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-model-cache\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.303182 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.303147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-home\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.305120 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.305086 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-dshm\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.305315 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.305297 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tls-certs\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.311656 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.311632 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtngq\" (UniqueName: \"kubernetes.io/projected/2f38e7a3-0a32-41f6-b51f-df2fcc370421-kube-api-access-vtngq\") pod \"e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.455736 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.455651 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:42.597017 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:42.596992 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9"] Jul 24 18:13:42.599355 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:13:42.599328 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f38e7a3_0a32_41f6_b51f_df2fcc370421.slice/crio-ccd37f39a41db5894c3e849297c933744fff32731b9ddfaf7e22e8a0f955f413 WatchSource:0}: Error finding container ccd37f39a41db5894c3e849297c933744fff32731b9ddfaf7e22e8a0f955f413: Status 404 returned error can't find the container with id ccd37f39a41db5894c3e849297c933744fff32731b9ddfaf7e22e8a0f955f413 Jul 24 18:13:43.008291 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:43.008253 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" event={"ID":"2f38e7a3-0a32-41f6-b51f-df2fcc370421","Type":"ContainerStarted","Data":"bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce"} Jul 24 18:13:43.008291 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:43.008297 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" event={"ID":"2f38e7a3-0a32-41f6-b51f-df2fcc370421","Type":"ContainerStarted","Data":"ccd37f39a41db5894c3e849297c933744fff32731b9ddfaf7e22e8a0f955f413"} Jul 24 18:13:43.026618 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:43.026566 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" podStartSLOduration=1.026551109 podStartE2EDuration="1.026551109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:13:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:13:43.025670037 +0000 UTC m=+1848.065464080" watchObservedRunningTime="2026-07-24 18:13:43.026551109 +0000 UTC m=+1848.066345128" Jul 24 18:13:44.456326 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:44.456291 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:13:52.455981 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:52.455943 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:52.456362 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:52.456000 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:52.468834 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:52.468811 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:53.065295 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:53.065267 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:13:58.755416 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:13:58.755377 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:14:02.402199 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.402163 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs"] Jul 24 18:14:02.405690 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.405667 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" Jul 24 18:14:02.408762 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.408743 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:14:02.408901 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.408746 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:14:02.417226 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.417206 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs"] Jul 24 18:14:02.487357 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.487323 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4865aa1-ec97-4604-83ce-9ef5b8028184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-159379b5-nlngs\" (UID: \"d4865aa1-ec97-4604-83ce-9ef5b8028184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" Jul 24 18:14:02.588072 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.588026 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4865aa1-ec97-4604-83ce-9ef5b8028184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-159379b5-nlngs\" (UID: \"d4865aa1-ec97-4604-83ce-9ef5b8028184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" Jul 24 18:14:02.588653 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.588630 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4865aa1-ec97-4604-83ce-9ef5b8028184-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-159379b5-nlngs\" (UID: \"d4865aa1-ec97-4604-83ce-9ef5b8028184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" Jul 24 18:14:02.715862 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.715750 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" Jul 24 18:14:02.855704 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:02.855652 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs"] Jul 24 18:14:02.863248 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:14:02.863216 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4865aa1_ec97_4604_83ce_9ef5b8028184.slice/crio-1fa58ef87dc67979ffe16eafb50ec7921df3627b1e2183371c691233ec880b77 WatchSource:0}: Error finding container 1fa58ef87dc67979ffe16eafb50ec7921df3627b1e2183371c691233ec880b77: Status 404 returned error can't find the container with id 1fa58ef87dc67979ffe16eafb50ec7921df3627b1e2183371c691233ec880b77 Jul 24 18:14:03.104245 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:03.104210 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" event={"ID":"d4865aa1-ec97-4604-83ce-9ef5b8028184","Type":"ContainerStarted","Data":"1fa58ef87dc67979ffe16eafb50ec7921df3627b1e2183371c691233ec880b77"} Jul 24 18:14:04.109702 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:04.109670 2559 generic.go:356] "Generic (PLEG): container finished" podID="d4865aa1-ec97-4604-83ce-9ef5b8028184" containerID="74db8b0e250e90c5544f41a2b674a4e70f0f0985f4378804055ed7cb4ee22bd2" exitCode=0 Jul 24 18:14:04.110100 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:04.109755 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" event={"ID":"d4865aa1-ec97-4604-83ce-9ef5b8028184","Type":"ContainerDied","Data":"74db8b0e250e90c5544f41a2b674a4e70f0f0985f4378804055ed7cb4ee22bd2"} Jul 24 18:14:05.247512 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:05.247487 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" Jul 24 18:14:05.412955 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:05.412853 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4865aa1-ec97-4604-83ce-9ef5b8028184-maas-api-service-ca\") pod \"d4865aa1-ec97-4604-83ce-9ef5b8028184\" (UID: \"d4865aa1-ec97-4604-83ce-9ef5b8028184\") " Jul 24 18:14:05.413266 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:05.413241 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4865aa1-ec97-4604-83ce-9ef5b8028184-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4865aa1-ec97-4604-83ce-9ef5b8028184" (UID: "d4865aa1-ec97-4604-83ce-9ef5b8028184"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:14:05.514133 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:05.514099 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4865aa1-ec97-4604-83ce-9ef5b8028184-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:14:06.119592 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:06.119560 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" event={"ID":"d4865aa1-ec97-4604-83ce-9ef5b8028184","Type":"ContainerDied","Data":"1fa58ef87dc67979ffe16eafb50ec7921df3627b1e2183371c691233ec880b77"} Jul 24 18:14:06.119772 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:06.119613 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1fa58ef87dc67979ffe16eafb50ec7921df3627b1e2183371c691233ec880b77" Jul 24 18:14:06.119772 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:06.119568 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs" Jul 24 18:14:15.352513 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:15.352478 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:14:21.061525 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:21.061445 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:14:22.469754 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:22.469720 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs"] Jul 24 18:14:22.472760 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:22.472734 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-159379b5-nlngs"] Jul 24 18:14:23.494187 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:23.494156 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4865aa1-ec97-4604-83ce-9ef5b8028184" path="/var/lib/kubelet/pods/d4865aa1-ec97-4604-83ce-9ef5b8028184/volumes" Jul 24 18:14:30.788576 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:30.788538 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9"] Jul 24 18:14:45.808123 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:45.808061 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" podUID="2f38e7a3-0a32-41f6-b51f-df2fcc370421" containerName="main" containerID="cri-o://bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce" gracePeriod=45 Jul 24 18:14:46.068558 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.068534 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:14:46.181224 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181195 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-home\") pod \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " Jul 24 18:14:46.181421 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181249 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tmp-dir\") pod \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " Jul 24 18:14:46.181421 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181301 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-model-cache\") pod \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " Jul 24 18:14:46.181421 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181349 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-dshm\") pod \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " Jul 24 18:14:46.181421 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181373 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tls-certs\") pod \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " Jul 24 18:14:46.181421 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181401 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtngq\" (UniqueName: \"kubernetes.io/projected/2f38e7a3-0a32-41f6-b51f-df2fcc370421-kube-api-access-vtngq\") pod \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\" (UID: \"2f38e7a3-0a32-41f6-b51f-df2fcc370421\") " Jul 24 18:14:46.181698 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181478 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-home" (OuterVolumeSpecName: "home") pod "2f38e7a3-0a32-41f6-b51f-df2fcc370421" (UID: "2f38e7a3-0a32-41f6-b51f-df2fcc370421"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:14:46.181698 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181586 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f38e7a3-0a32-41f6-b51f-df2fcc370421" (UID: "2f38e7a3-0a32-41f6-b51f-df2fcc370421"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:14:46.181698 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181604 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f38e7a3-0a32-41f6-b51f-df2fcc370421" (UID: "2f38e7a3-0a32-41f6-b51f-df2fcc370421"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:14:46.181916 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181708 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:14:46.181916 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181725 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:14:46.181916 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.181742 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:14:46.183934 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.183906 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f38e7a3-0a32-41f6-b51f-df2fcc370421-kube-api-access-vtngq" (OuterVolumeSpecName: "kube-api-access-vtngq") pod "2f38e7a3-0a32-41f6-b51f-df2fcc370421" (UID: "2f38e7a3-0a32-41f6-b51f-df2fcc370421"). InnerVolumeSpecName "kube-api-access-vtngq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:14:46.183934 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.183921 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-dshm" (OuterVolumeSpecName: "dshm") pod "2f38e7a3-0a32-41f6-b51f-df2fcc370421" (UID: "2f38e7a3-0a32-41f6-b51f-df2fcc370421"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:14:46.183934 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.183925 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f38e7a3-0a32-41f6-b51f-df2fcc370421" (UID: "2f38e7a3-0a32-41f6-b51f-df2fcc370421"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:14:46.283074 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.283036 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f38e7a3-0a32-41f6-b51f-df2fcc370421-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:14:46.283074 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.283065 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f38e7a3-0a32-41f6-b51f-df2fcc370421-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:14:46.283074 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.283076 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtngq\" (UniqueName: \"kubernetes.io/projected/2f38e7a3-0a32-41f6-b51f-df2fcc370421-kube-api-access-vtngq\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:14:46.305592 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.305552 2559 generic.go:356] "Generic (PLEG): container finished" podID="2f38e7a3-0a32-41f6-b51f-df2fcc370421" containerID="bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce" exitCode=0 Jul 24 18:14:46.305592 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.305595 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" Jul 24 18:14:46.305813 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.305632 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" event={"ID":"2f38e7a3-0a32-41f6-b51f-df2fcc370421","Type":"ContainerDied","Data":"bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce"} Jul 24 18:14:46.305813 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.305669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9" event={"ID":"2f38e7a3-0a32-41f6-b51f-df2fcc370421","Type":"ContainerDied","Data":"ccd37f39a41db5894c3e849297c933744fff32731b9ddfaf7e22e8a0f955f413"} Jul 24 18:14:46.305813 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.305692 2559 scope.go:117] "RemoveContainer" containerID="bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce" Jul 24 18:14:46.314797 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.314764 2559 scope.go:117] "RemoveContainer" containerID="bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce" Jul 24 18:14:46.315060 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:14:46.315039 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce\": container with ID starting with bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce not found: ID does not exist" containerID="bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce" Jul 24 18:14:46.315123 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.315073 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce"} err="failed to get container status \"bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce\": rpc error: code = NotFound desc = could not find container \"bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce\": container with ID starting with bb2f1c44cd5de59cfdf71d10281c3472ac13a8ccb88afb1c83b33b2f5561e2ce not found: ID does not exist" Jul 24 18:14:46.327062 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.327005 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9"] Jul 24 18:14:46.330384 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:46.330361 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-159379b5/e2e-lifecycle-model-159379b5-kserve-74fd77ff6c-zs6t9"] Jul 24 18:14:47.494214 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:47.494181 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f38e7a3-0a32-41f6-b51f-df2fcc370421" path="/var/lib/kubelet/pods/2f38e7a3-0a32-41f6-b51f-df2fcc370421/volumes" Jul 24 18:14:55.600219 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:14:55.600183 2559 scope.go:117] "RemoveContainer" containerID="4a1e702752a2cfbd640696e535735b4dbf5bb6ee8d394f63e8c2c096381e0191" Jul 24 18:15:00.150028 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.149998 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr"] Jul 24 18:15:00.150454 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.150346 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f38e7a3-0a32-41f6-b51f-df2fcc370421" containerName="main" Jul 24 18:15:00.150454 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.150359 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f38e7a3-0a32-41f6-b51f-df2fcc370421" containerName="main" Jul 24 18:15:00.150454 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.150373 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4865aa1-ec97-4604-83ce-9ef5b8028184" containerName="revoke-keys" Jul 24 18:15:00.150454 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.150378 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4865aa1-ec97-4604-83ce-9ef5b8028184" containerName="revoke-keys" Jul 24 18:15:00.150454 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.150441 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f38e7a3-0a32-41f6-b51f-df2fcc370421" containerName="main" Jul 24 18:15:00.150616 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.150456 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4865aa1-ec97-4604-83ce-9ef5b8028184" containerName="revoke-keys" Jul 24 18:15:00.153592 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.153572 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr" Jul 24 18:15:00.157326 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.157307 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:15:00.163914 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.163870 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr" Jul 24 18:15:00.164991 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.164969 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr"] Jul 24 18:15:00.303951 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.303909 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr"] Jul 24 18:15:00.306730 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:15:00.306705 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ef608b7_af6d_4bfc_bdd4_8e2c38cac8d0.slice/crio-d449697461a9bd5b48b31fa74325e5753519e252b6d4ed19760f97e8a92b2321 WatchSource:0}: Error finding container d449697461a9bd5b48b31fa74325e5753519e252b6d4ed19760f97e8a92b2321: Status 404 returned error can't find the container with id d449697461a9bd5b48b31fa74325e5753519e252b6d4ed19760f97e8a92b2321 Jul 24 18:15:00.377393 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:00.377360 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr" event={"ID":"4ef608b7-af6d-4bfc-bdd4-8e2c38cac8d0","Type":"ContainerStarted","Data":"d449697461a9bd5b48b31fa74325e5753519e252b6d4ed19760f97e8a92b2321"} Jul 24 18:15:01.389345 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:01.389237 2559 generic.go:356] "Generic (PLEG): container finished" podID="4ef608b7-af6d-4bfc-bdd4-8e2c38cac8d0" containerID="7b64e329b4e8343746c6c4f1e13cba32b56480df3728d3db12513044e5dc76b8" exitCode=0 Jul 24 18:15:01.389345 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:01.389327 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr" event={"ID":"4ef608b7-af6d-4bfc-bdd4-8e2c38cac8d0","Type":"ContainerDied","Data":"7b64e329b4e8343746c6c4f1e13cba32b56480df3728d3db12513044e5dc76b8"} Jul 24 18:15:02.527005 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:02.526983 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr" Jul 24 18:15:03.400019 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:03.399978 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr" event={"ID":"4ef608b7-af6d-4bfc-bdd4-8e2c38cac8d0","Type":"ContainerDied","Data":"d449697461a9bd5b48b31fa74325e5753519e252b6d4ed19760f97e8a92b2321"} Jul 24 18:15:03.400200 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:03.400043 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d449697461a9bd5b48b31fa74325e5753519e252b6d4ed19760f97e8a92b2321" Jul 24 18:15:03.400200 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:03.399993 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748615-nf6qr" Jul 24 18:15:21.886420 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:21.886384 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:15:30.469274 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:30.469240 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:15:55.623596 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:15:55.623565 2559 scope.go:117] "RemoveContainer" containerID="a6dd6373eceb6ff3f6f4fe9148664a6e59ece93098b69037bf6df2cb6707c1d5" Jul 24 18:16:17.573886 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:17.573848 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:16:19.905927 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.905894 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx"] Jul 24 18:16:19.906299 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.906232 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ef608b7-af6d-4bfc-bdd4-8e2c38cac8d0" containerName="cleanup" Jul 24 18:16:19.906299 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.906243 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ef608b7-af6d-4bfc-bdd4-8e2c38cac8d0" containerName="cleanup" Jul 24 18:16:19.906398 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.906302 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ef608b7-af6d-4bfc-bdd4-8e2c38cac8d0" containerName="cleanup" Jul 24 18:16:19.909646 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.909627 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.912384 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.912363 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dd7b67-openshift-default-dockercfg-xv9ph\"" Jul 24 18:16:19.923350 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.923323 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx"] Jul 24 18:16:19.980681 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980645 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-credential-socket\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.980681 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980682 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-data\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.980962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980713 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d32026-26c2-40ad-9682-185d70fdc79b-istiod-ca-cert\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.980962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980819 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qsbv\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-kube-api-access-6qsbv\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.980962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980864 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-podinfo\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.980962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980905 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-certs\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.980962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980927 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-socket\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.980962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980942 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-envoy\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:19.980962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:19.980959 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-token\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.081743 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.081703 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-data\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.081743 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.081747 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d32026-26c2-40ad-9682-185d70fdc79b-istiod-ca-cert\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082008 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.081774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qsbv\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-kube-api-access-6qsbv\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082008 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.081812 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-podinfo\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082008 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.081921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-certs\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082008 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.081972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-socket\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082008 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.081996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-envoy\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082264 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.082018 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-token\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082264 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.082100 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-credential-socket\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082264 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.082145 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-data\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082264 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.082241 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-certs\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082467 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.082280 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-socket\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082467 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.082398 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-credential-socket\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.082550 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.082506 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d32026-26c2-40ad-9682-185d70fdc79b-istiod-ca-cert\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.084324 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.084296 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-envoy\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.084512 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.084495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-podinfo\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.091099 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.091071 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-token\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.091309 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.091291 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qsbv\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-kube-api-access-6qsbv\") pod \"e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.220123 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.220038 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:20.364299 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.364274 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx"] Jul 24 18:16:20.366825 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:16:20.366773 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d32026_26c2_40ad_9682_185d70fdc79b.slice/crio-9947d13e67a8085f6bc2af64dfed21214dd758a50af98cca7fc1188042d45761 WatchSource:0}: Error finding container 9947d13e67a8085f6bc2af64dfed21214dd758a50af98cca7fc1188042d45761: Status 404 returned error can't find the container with id 9947d13e67a8085f6bc2af64dfed21214dd758a50af98cca7fc1188042d45761 Jul 24 18:16:20.369144 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.369115 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:16:20.369220 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.369180 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:16:20.369220 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.369211 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:16:20.761808 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.761754 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" event={"ID":"b7d32026-26c2-40ad-9682-185d70fdc79b","Type":"ContainerStarted","Data":"4835b6f3d6f1150568922482bb84dc6b56f58ea84f084916ba167ae58abebda3"} Jul 24 18:16:20.761998 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.761844 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" event={"ID":"b7d32026-26c2-40ad-9682-185d70fdc79b","Type":"ContainerStarted","Data":"9947d13e67a8085f6bc2af64dfed21214dd758a50af98cca7fc1188042d45761"} Jul 24 18:16:20.789101 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:20.789043 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" podStartSLOduration=1.78902172 podStartE2EDuration="1.78902172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:20.78606972 +0000 UTC m=+2005.825863761" watchObservedRunningTime="2026-07-24 18:16:20.78902172 +0000 UTC m=+2005.828815761" Jul 24 18:16:21.220611 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:21.220576 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:21.225455 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:21.225431 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:21.767136 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:21.767107 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:21.768064 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:21.768044 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:26.556617 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:26.556585 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:16:35.849888 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:35.849852 2559 generic.go:356] "Generic (PLEG): container finished" podID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerID="4835b6f3d6f1150568922482bb84dc6b56f58ea84f084916ba167ae58abebda3" exitCode=0 Jul 24 18:16:35.850281 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:35.849920 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" event={"ID":"b7d32026-26c2-40ad-9682-185d70fdc79b","Type":"ContainerDied","Data":"4835b6f3d6f1150568922482bb84dc6b56f58ea84f084916ba167ae58abebda3"} Jul 24 18:16:35.850322 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:35.850303 2559 scope.go:117] "RemoveContainer" containerID="4835b6f3d6f1150568922482bb84dc6b56f58ea84f084916ba167ae58abebda3" Jul 24 18:16:35.851388 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:35.851356 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:16:35.851626 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:35.851442 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:16:35.851626 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:35.851483 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:16:36.767039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:36.767005 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:36.855956 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:36.855924 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" event={"ID":"b7d32026-26c2-40ad-9682-185d70fdc79b","Type":"ContainerStarted","Data":"c56c784bb2edb9fce343357d48305c33478ebd4ea8005f4acd2ca7c5b13cee18"} Jul 24 18:16:37.221147 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:37.221106 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:37.222157 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:37.222133 2559 patch_prober.go:28] interesting pod/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 24 18:16:37.222294 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:37.222179 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 24 18:16:38.221310 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:38.221279 2559 patch_prober.go:28] interesting pod/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 24 18:16:38.221729 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:38.221335 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 24 18:16:39.220932 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:39.220885 2559 patch_prober.go:28] interesting pod/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 24 18:16:39.221131 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:39.220952 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 24 18:16:40.220447 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.220409 2559 patch_prober.go:28] interesting pod/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 24 18:16:40.220897 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.220467 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 24 18:16:40.748286 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.748251 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv"] Jul 24 18:16:40.751743 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.751721 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.754372 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.754354 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04ecb2\"/\"default-dockercfg-bgftd\"" Jul 24 18:16:40.755485 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.755465 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-04ecb2\"/\"test-model-04ecb2-kserve-self-signed-certs\"" Jul 24 18:16:40.755601 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.755549 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04ecb2\"/\"kube-root-ca.crt\"" Jul 24 18:16:40.755653 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.755615 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-04ecb2\"/\"openshift-service-ca.crt\"" Jul 24 18:16:40.761194 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.761171 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv"] Jul 24 18:16:40.873751 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.873717 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-tmp-dir\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.873950 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.873864 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-model-cache\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.873950 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.873897 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb64b\" (UniqueName: \"kubernetes.io/projected/3d4104e5-39d0-46aa-bd3d-6b975333e239-kube-api-access-rb64b\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.873950 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.873928 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-dshm\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.874126 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.873950 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4104e5-39d0-46aa-bd3d-6b975333e239-tls-certs\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.874126 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.874083 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-home\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975313 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975280 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-model-cache\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975313 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975321 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb64b\" (UniqueName: \"kubernetes.io/projected/3d4104e5-39d0-46aa-bd3d-6b975333e239-kube-api-access-rb64b\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975588 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975342 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-dshm\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975588 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975360 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4104e5-39d0-46aa-bd3d-6b975333e239-tls-certs\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975588 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975433 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-home\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975588 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975464 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-tmp-dir\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975836 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975772 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-model-cache\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975932 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975906 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-home\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.975995 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.975916 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-tmp-dir\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.978591 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.978562 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-dshm\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.979272 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.979245 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4104e5-39d0-46aa-bd3d-6b975333e239-tls-certs\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:40.986632 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:40.986604 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb64b\" (UniqueName: \"kubernetes.io/projected/3d4104e5-39d0-46aa-bd3d-6b975333e239-kube-api-access-rb64b\") pod \"test-model-04ecb2-kserve-dfd8877b5-pn8qv\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:41.061708 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:41.061683 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:41.220494 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:41.220453 2559 patch_prober.go:28] interesting pod/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" start-of-body= Jul 24 18:16:41.221050 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:41.221024 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.41:15021/healthz/ready\": dial tcp 10.132.0.41:15021: connect: connection refused" Jul 24 18:16:41.250632 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:41.250606 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv"] Jul 24 18:16:41.252904 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:16:41.252877 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3d4104e5_39d0_46aa_bd3d_6b975333e239.slice/crio-3b7e9801154f3a73a7b1b10b046cb08e3e3ad8378a5967e732d86c6e3f755869 WatchSource:0}: Error finding container 3b7e9801154f3a73a7b1b10b046cb08e3e3ad8378a5967e732d86c6e3f755869: Status 404 returned error can't find the container with id 3b7e9801154f3a73a7b1b10b046cb08e3e3ad8378a5967e732d86c6e3f755869 Jul 24 18:16:41.880692 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:41.880656 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" event={"ID":"3d4104e5-39d0-46aa-bd3d-6b975333e239","Type":"ContainerStarted","Data":"bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46"} Jul 24 18:16:41.880692 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:41.880700 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" event={"ID":"3d4104e5-39d0-46aa-bd3d-6b975333e239","Type":"ContainerStarted","Data":"3b7e9801154f3a73a7b1b10b046cb08e3e3ad8378a5967e732d86c6e3f755869"} Jul 24 18:16:41.900853 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:41.900760 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" podStartSLOduration=1.900736593 podStartE2EDuration="1.900736593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:16:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:16:41.899396389 +0000 UTC m=+2026.939190442" watchObservedRunningTime="2026-07-24 18:16:41.900736593 +0000 UTC m=+2026.940530635" Jul 24 18:16:42.224645 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:42.224574 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:42.225038 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:42.224855 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:42.225481 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:42.225461 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:16:43.256733 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:43.256698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:16:50.457092 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:50.457053 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:16:51.062601 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:51.062571 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:51.062840 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:51.062644 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:51.075700 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:51.075672 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:51.941622 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:51.941596 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:16:55.643083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:16:55.643048 2559 scope.go:117] "RemoveContainer" containerID="9deb00a47ccaf9dedfc39e9586f7776938ef33ae4e54474d6fea7172fea6557f" Jul 24 18:17:09.855083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:17:09.855046 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:17:19.462822 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:17:19.462717 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:17:52.573067 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:17:52.573035 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv"] Jul 24 18:17:55.584523 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:17:55.584485 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:17:55.585742 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:17:55.585718 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:17:56.266953 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:17:56.266920 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:18:06.355296 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:06.355259 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:18:07.593760 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.593698 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" podUID="3d4104e5-39d0-46aa-bd3d-6b975333e239" containerName="main" containerID="cri-o://bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46" gracePeriod=45 Jul 24 18:18:07.859552 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.859522 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:18:07.948866 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.948834 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb64b\" (UniqueName: \"kubernetes.io/projected/3d4104e5-39d0-46aa-bd3d-6b975333e239-kube-api-access-rb64b\") pod \"3d4104e5-39d0-46aa-bd3d-6b975333e239\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " Jul 24 18:18:07.949036 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.948877 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-tmp-dir\") pod \"3d4104e5-39d0-46aa-bd3d-6b975333e239\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " Jul 24 18:18:07.949036 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.948898 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-home\") pod \"3d4104e5-39d0-46aa-bd3d-6b975333e239\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " Jul 24 18:18:07.949036 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.948969 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-model-cache\") pod \"3d4104e5-39d0-46aa-bd3d-6b975333e239\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " Jul 24 18:18:07.949036 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.949004 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4104e5-39d0-46aa-bd3d-6b975333e239-tls-certs\") pod \"3d4104e5-39d0-46aa-bd3d-6b975333e239\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " Jul 24 18:18:07.949249 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.949047 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-dshm\") pod \"3d4104e5-39d0-46aa-bd3d-6b975333e239\" (UID: \"3d4104e5-39d0-46aa-bd3d-6b975333e239\") " Jul 24 18:18:07.949249 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.949174 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3d4104e5-39d0-46aa-bd3d-6b975333e239" (UID: "3d4104e5-39d0-46aa-bd3d-6b975333e239"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:07.949249 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.949215 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-model-cache" (OuterVolumeSpecName: "model-cache") pod "3d4104e5-39d0-46aa-bd3d-6b975333e239" (UID: "3d4104e5-39d0-46aa-bd3d-6b975333e239"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:07.949249 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.949233 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-home" (OuterVolumeSpecName: "home") pod "3d4104e5-39d0-46aa-bd3d-6b975333e239" (UID: "3d4104e5-39d0-46aa-bd3d-6b975333e239"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:07.949483 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.949324 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:07.949483 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.949341 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:07.949483 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.949353 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:07.951255 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.951227 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-dshm" (OuterVolumeSpecName: "dshm") pod "3d4104e5-39d0-46aa-bd3d-6b975333e239" (UID: "3d4104e5-39d0-46aa-bd3d-6b975333e239"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:07.951363 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.951305 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3d4104e5-39d0-46aa-bd3d-6b975333e239-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3d4104e5-39d0-46aa-bd3d-6b975333e239" (UID: "3d4104e5-39d0-46aa-bd3d-6b975333e239"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:18:07.951363 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:07.951345 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d4104e5-39d0-46aa-bd3d-6b975333e239-kube-api-access-rb64b" (OuterVolumeSpecName: "kube-api-access-rb64b") pod "3d4104e5-39d0-46aa-bd3d-6b975333e239" (UID: "3d4104e5-39d0-46aa-bd3d-6b975333e239"). InnerVolumeSpecName "kube-api-access-rb64b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:18:08.050356 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.050308 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3d4104e5-39d0-46aa-bd3d-6b975333e239-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:08.050356 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.050350 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3d4104e5-39d0-46aa-bd3d-6b975333e239-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:08.050356 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.050359 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb64b\" (UniqueName: \"kubernetes.io/projected/3d4104e5-39d0-46aa-bd3d-6b975333e239-kube-api-access-rb64b\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:08.297449 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.297360 2559 generic.go:356] "Generic (PLEG): container finished" podID="3d4104e5-39d0-46aa-bd3d-6b975333e239" containerID="bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46" exitCode=0 Jul 24 18:18:08.297449 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.297411 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" Jul 24 18:18:08.297661 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.297451 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" event={"ID":"3d4104e5-39d0-46aa-bd3d-6b975333e239","Type":"ContainerDied","Data":"bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46"} Jul 24 18:18:08.297661 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.297495 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv" event={"ID":"3d4104e5-39d0-46aa-bd3d-6b975333e239","Type":"ContainerDied","Data":"3b7e9801154f3a73a7b1b10b046cb08e3e3ad8378a5967e732d86c6e3f755869"} Jul 24 18:18:08.297661 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.297513 2559 scope.go:117] "RemoveContainer" containerID="bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46" Jul 24 18:18:08.306870 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.306851 2559 scope.go:117] "RemoveContainer" containerID="bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46" Jul 24 18:18:08.307147 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:18:08.307127 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46\": container with ID starting with bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46 not found: ID does not exist" containerID="bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46" Jul 24 18:18:08.307198 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.307155 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46"} err="failed to get container status \"bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46\": rpc error: code = NotFound desc = could not find container \"bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46\": container with ID starting with bc4c0512945ead31bce5a5bfaed89ef66d3495a6a3a3d13f712ad21bc23edd46 not found: ID does not exist" Jul 24 18:18:08.317290 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.317263 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv"] Jul 24 18:18:08.320982 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:08.320962 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-04ecb2/test-model-04ecb2-kserve-dfd8877b5-pn8qv"] Jul 24 18:18:09.494575 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:09.494543 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d4104e5-39d0-46aa-bd3d-6b975333e239" path="/var/lib/kubelet/pods/3d4104e5-39d0-46aa-bd3d-6b975333e239/volumes" Jul 24 18:18:28.254383 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:28.254351 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:18:37.282120 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:37.282085 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx"] Jul 24 18:18:37.282648 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:37.282364 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" containerID="cri-o://c56c784bb2edb9fce343357d48305c33478ebd4ea8005f4acd2ca7c5b13cee18" gracePeriod=30 Jul 24 18:18:42.226960 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.226927 2559 patch_prober.go:28] interesting pod/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 18:18:42.227344 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.226983 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 18:18:42.462103 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.462074 2559 generic.go:356] "Generic (PLEG): container finished" podID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerID="c56c784bb2edb9fce343357d48305c33478ebd4ea8005f4acd2ca7c5b13cee18" exitCode=0 Jul 24 18:18:42.462269 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.462146 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" event={"ID":"b7d32026-26c2-40ad-9682-185d70fdc79b","Type":"ContainerDied","Data":"c56c784bb2edb9fce343357d48305c33478ebd4ea8005f4acd2ca7c5b13cee18"} Jul 24 18:18:42.462269 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.462224 2559 scope.go:117] "RemoveContainer" containerID="4835b6f3d6f1150568922482bb84dc6b56f58ea84f084916ba167ae58abebda3" Jul 24 18:18:42.547818 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.547774 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:18:42.656723 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.656690 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-envoy\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.656980 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.656962 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d32026-26c2-40ad-9682-185d70fdc79b-istiod-ca-cert\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.657069 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657026 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-data\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.657069 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657053 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-podinfo\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.657182 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657077 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-token\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.657182 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657104 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qsbv\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-kube-api-access-6qsbv\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.657182 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657146 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-credential-socket\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.657182 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657177 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-certs\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.657379 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657230 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-socket\") pod \"b7d32026-26c2-40ad-9682-185d70fdc79b\" (UID: \"b7d32026-26c2-40ad-9682-185d70fdc79b\") " Jul 24 18:18:42.657482 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657453 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:42.657587 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657564 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7d32026-26c2-40ad-9682-185d70fdc79b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:18:42.657745 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657714 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:42.657964 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.657944 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:42.658174 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.658150 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:42.659587 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.659560 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:18:42.660004 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.659976 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:18:42.660258 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.660234 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:18:42.660430 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.660406 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-kube-api-access-6qsbv" (OuterVolumeSpecName: "kube-api-access-6qsbv") pod "b7d32026-26c2-40ad-9682-185d70fdc79b" (UID: "b7d32026-26c2-40ad-9682-185d70fdc79b"). InnerVolumeSpecName "kube-api-access-6qsbv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:18:42.758383 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758344 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-data\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:42.758383 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758376 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-podinfo\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:42.758383 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758386 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-token\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:42.758626 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758394 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qsbv\" (UniqueName: \"kubernetes.io/projected/b7d32026-26c2-40ad-9682-185d70fdc79b-kube-api-access-6qsbv\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:42.758626 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758405 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-credential-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:42.758626 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758413 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:42.758626 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758421 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-workload-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:42.758626 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758429 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7d32026-26c2-40ad-9682-185d70fdc79b-istio-envoy\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:42.758626 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:42.758436 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7d32026-26c2-40ad-9682-185d70fdc79b-istiod-ca-cert\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:18:43.468204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:43.468177 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" Jul 24 18:18:43.468658 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:43.468176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx" event={"ID":"b7d32026-26c2-40ad-9682-185d70fdc79b","Type":"ContainerDied","Data":"9947d13e67a8085f6bc2af64dfed21214dd758a50af98cca7fc1188042d45761"} Jul 24 18:18:43.468658 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:43.468307 2559 scope.go:117] "RemoveContainer" containerID="c56c784bb2edb9fce343357d48305c33478ebd4ea8005f4acd2ca7c5b13cee18" Jul 24 18:18:43.499591 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:43.499557 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx"] Jul 24 18:18:43.510488 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:43.510456 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dd7b67-openshift-default-6768fd89fb-78ppx"] Jul 24 18:18:45.493438 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:45.493406 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" path="/var/lib/kubelet/pods/b7d32026-26c2-40ad-9682-185d70fdc79b/volumes" Jul 24 18:18:45.651505 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:45.651474 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:18:46.953679 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.953583 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f"] Jul 24 18:18:46.954083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.953970 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" Jul 24 18:18:46.954083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.953982 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" Jul 24 18:18:46.954083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.954015 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" Jul 24 18:18:46.954083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.954022 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" Jul 24 18:18:46.954083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.954037 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d4104e5-39d0-46aa-bd3d-6b975333e239" containerName="main" Jul 24 18:18:46.954083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.954043 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d4104e5-39d0-46aa-bd3d-6b975333e239" containerName="main" Jul 24 18:18:46.954275 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.954099 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" Jul 24 18:18:46.954275 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.954107 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7d32026-26c2-40ad-9682-185d70fdc79b" containerName="istio-proxy" Jul 24 18:18:46.954275 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.954114 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d4104e5-39d0-46aa-bd3d-6b975333e239" containerName="main" Jul 24 18:18:46.958614 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.958591 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:46.961485 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.961462 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-ac2de9-openshift-default-dockercfg-79l9x\"" Jul 24 18:18:46.972765 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:46.972739 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f"] Jul 24 18:18:47.097596 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097555 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-credential-socket\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.097771 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097651 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-certs\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.097771 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097683 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-token\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.097771 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a6078e-3aaa-4950-9032-3193b1ec10f5-istiod-ca-cert\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.097771 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097767 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-envoy\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.097954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097830 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-data\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.097954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097865 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l78r7\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-kube-api-access-l78r7\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.097954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097920 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-socket\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.097954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.097943 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-podinfo\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.198976 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.198930 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-certs\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.198976 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.198972 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-token\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199196 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.198990 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a6078e-3aaa-4950-9032-3193b1ec10f5-istiod-ca-cert\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199196 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199110 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-envoy\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199196 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199159 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-data\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199338 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199200 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l78r7\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-kube-api-access-l78r7\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199338 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199252 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-socket\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199338 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-podinfo\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199483 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199345 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-credential-socket\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199483 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199395 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-certs\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199591 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199564 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-socket\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199646 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199615 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-data\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.199811 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.199764 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-credential-socket\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.200033 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.200010 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a6078e-3aaa-4950-9032-3193b1ec10f5-istiod-ca-cert\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.201565 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.201530 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-envoy\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.201863 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.201848 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-podinfo\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.208871 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.208774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-token\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.209076 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.209052 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l78r7\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-kube-api-access-l78r7\") pod \"e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.268893 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.268841 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:47.418535 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.418510 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f"] Jul 24 18:18:47.421056 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:18:47.421021 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a6078e_3aaa_4950_9032_3193b1ec10f5.slice/crio-f3fa517b3b4b22717f0049824eaf11064caf269ccf905b9ca2cbba703f765373 WatchSource:0}: Error finding container f3fa517b3b4b22717f0049824eaf11064caf269ccf905b9ca2cbba703f765373: Status 404 returned error can't find the container with id f3fa517b3b4b22717f0049824eaf11064caf269ccf905b9ca2cbba703f765373 Jul 24 18:18:47.423218 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.423190 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:18:47.423322 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.423256 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:18:47.423322 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.423284 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:18:47.494246 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.494209 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" event={"ID":"39a6078e-3aaa-4950-9032-3193b1ec10f5","Type":"ContainerStarted","Data":"f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441"} Jul 24 18:18:47.494435 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.494272 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" event={"ID":"39a6078e-3aaa-4950-9032-3193b1ec10f5","Type":"ContainerStarted","Data":"f3fa517b3b4b22717f0049824eaf11064caf269ccf905b9ca2cbba703f765373"} Jul 24 18:18:47.526592 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:47.526501 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" podStartSLOduration=1.5264778460000001 podStartE2EDuration="1.526477846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:18:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:18:47.525862215 +0000 UTC m=+2152.565656258" watchObservedRunningTime="2026-07-24 18:18:47.526477846 +0000 UTC m=+2152.566271868" Jul 24 18:18:48.269427 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:48.269389 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:48.274557 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:48.274530 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:48.496453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:48.496419 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:48.497615 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:48.497598 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:18:53.551759 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:53.551730 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:18:55.669431 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:18:55.669386 2559 scope.go:117] "RemoveContainer" containerID="e1f9b15c660dddc2f47a3e7d29cb4b8b04964df9b32790fed37f040cfe8959ba" Jul 24 18:19:02.190954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.190918 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh"] Jul 24 18:19:02.194476 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.194452 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" Jul 24 18:19:02.197427 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.197402 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:19:02.197573 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.197402 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:19:02.203843 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.203816 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh"] Jul 24 18:19:02.240576 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.240533 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c3743b-3069-4517-943d-4383eb8a5551-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh\" (UID: \"c8c3743b-3069-4517-943d-4383eb8a5551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" Jul 24 18:19:02.341632 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.341591 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c3743b-3069-4517-943d-4383eb8a5551-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh\" (UID: \"c8c3743b-3069-4517-943d-4383eb8a5551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" Jul 24 18:19:02.342229 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.342209 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c3743b-3069-4517-943d-4383eb8a5551-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh\" (UID: \"c8c3743b-3069-4517-943d-4383eb8a5551\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" Jul 24 18:19:02.507205 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.507113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" Jul 24 18:19:02.651776 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:02.651750 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh"] Jul 24 18:19:02.655018 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:19:02.654986 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c3743b_3069_4517_943d_4383eb8a5551.slice/crio-a5fce3d693b75df0b5dff55be31dd6314218efcf01ac7cdfd605586fc0bb2c5a WatchSource:0}: Error finding container a5fce3d693b75df0b5dff55be31dd6314218efcf01ac7cdfd605586fc0bb2c5a: Status 404 returned error can't find the container with id a5fce3d693b75df0b5dff55be31dd6314218efcf01ac7cdfd605586fc0bb2c5a Jul 24 18:19:03.498520 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.498442 2559 patch_prober.go:28] interesting pod/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 18:19:03.498520 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.498493 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 18:19:03.574609 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.574577 2559 generic.go:356] "Generic (PLEG): container finished" podID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerID="f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441" exitCode=0 Jul 24 18:19:03.574780 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.574650 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" event={"ID":"39a6078e-3aaa-4950-9032-3193b1ec10f5","Type":"ContainerDied","Data":"f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441"} Jul 24 18:19:03.575171 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.575153 2559 scope.go:117] "RemoveContainer" containerID="f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441" Jul 24 18:19:03.576290 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.576257 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:19:03.576379 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.576283 2559 generic.go:356] "Generic (PLEG): container finished" podID="c8c3743b-3069-4517-943d-4383eb8a5551" containerID="6333ac0959ebb5f24f12ccfe6f7e124accea58b4c6ea506ce4adc3498cea8238" exitCode=0 Jul 24 18:19:03.576379 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.576333 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:19:03.576379 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.576343 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" event={"ID":"c8c3743b-3069-4517-943d-4383eb8a5551","Type":"ContainerDied","Data":"6333ac0959ebb5f24f12ccfe6f7e124accea58b4c6ea506ce4adc3498cea8238"} Jul 24 18:19:03.576379 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.576369 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:19:03.576379 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:03.576379 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" event={"ID":"c8c3743b-3069-4517-943d-4383eb8a5551","Type":"ContainerStarted","Data":"a5fce3d693b75df0b5dff55be31dd6314218efcf01ac7cdfd605586fc0bb2c5a"} Jul 24 18:19:04.583457 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:04.583425 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" event={"ID":"39a6078e-3aaa-4950-9032-3193b1ec10f5","Type":"ContainerStarted","Data":"4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1"} Jul 24 18:19:04.763725 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:04.763701 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" Jul 24 18:19:04.864832 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:04.864706 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c3743b-3069-4517-943d-4383eb8a5551-maas-api-service-ca\") pod \"c8c3743b-3069-4517-943d-4383eb8a5551\" (UID: \"c8c3743b-3069-4517-943d-4383eb8a5551\") " Jul 24 18:19:04.865166 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:04.865135 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c3743b-3069-4517-943d-4383eb8a5551-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8c3743b-3069-4517-943d-4383eb8a5551" (UID: "c8c3743b-3069-4517-943d-4383eb8a5551"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:19:04.965494 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:04.965466 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c3743b-3069-4517-943d-4383eb8a5551-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:19:05.269290 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:05.269195 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:19:05.269450 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:05.269404 2559 patch_prober.go:28] interesting pod/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 18:19:05.269499 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:05.269441 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 18:19:05.591850 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:05.591815 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" Jul 24 18:19:05.592371 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:05.592244 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh" event={"ID":"c8c3743b-3069-4517-943d-4383eb8a5551","Type":"ContainerDied","Data":"a5fce3d693b75df0b5dff55be31dd6314218efcf01ac7cdfd605586fc0bb2c5a"} Jul 24 18:19:05.592371 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:05.592288 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a5fce3d693b75df0b5dff55be31dd6314218efcf01ac7cdfd605586fc0bb2c5a" Jul 24 18:19:06.269613 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:06.269572 2559 patch_prober.go:28] interesting pod/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 18:19:06.269819 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:06.269651 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 18:19:07.269959 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:07.269921 2559 patch_prober.go:28] interesting pod/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 18:19:07.270438 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:07.269998 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 18:19:08.269471 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:08.269434 2559 patch_prober.go:28] interesting pod/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 18:19:08.269731 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:08.269490 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 18:19:09.269250 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:09.269212 2559 patch_prober.go:28] interesting pod/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 24 18:19:09.269651 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:09.269267 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 24 18:19:10.273413 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:10.273385 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:19:10.273812 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:10.273642 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:19:10.274433 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:10.274418 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:19:20.468893 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:20.468856 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:19:22.271877 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:22.271836 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh"] Jul 24 18:19:22.274080 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:22.274057 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-ab4b1a-rmxnh"] Jul 24 18:19:23.493362 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:23.493327 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c3743b-3069-4517-943d-4383eb8a5551" path="/var/lib/kubelet/pods/c8c3743b-3069-4517-943d-4383eb8a5551/volumes" Jul 24 18:19:29.146817 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.146767 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq"] Jul 24 18:19:29.147282 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.147128 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c3743b-3069-4517-943d-4383eb8a5551" containerName="revoke-keys" Jul 24 18:19:29.147282 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.147139 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c3743b-3069-4517-943d-4383eb8a5551" containerName="revoke-keys" Jul 24 18:19:29.147282 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.147213 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c3743b-3069-4517-943d-4383eb8a5551" containerName="revoke-keys" Jul 24 18:19:29.151352 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.151335 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" Jul 24 18:19:29.154060 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.154041 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:19:29.154195 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.154042 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:19:29.157710 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.157680 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq"] Jul 24 18:19:29.186766 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.186736 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b758bb-0010-4f24-8040-132d81456f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq\" (UID: \"c8b758bb-0010-4f24-8040-132d81456f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" Jul 24 18:19:29.288133 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.288092 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b758bb-0010-4f24-8040-132d81456f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq\" (UID: \"c8b758bb-0010-4f24-8040-132d81456f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" Jul 24 18:19:29.288713 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.288695 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b758bb-0010-4f24-8040-132d81456f43-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq\" (UID: \"c8b758bb-0010-4f24-8040-132d81456f43\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" Jul 24 18:19:29.464624 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.464526 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" Jul 24 18:19:29.605585 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.605552 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq"] Jul 24 18:19:29.609231 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:19:29.609206 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b758bb_0010_4f24_8040_132d81456f43.slice/crio-b181b772c5e3cd89a439d9a94f8d89464761441d95eca40fe7e72290a97290c0 WatchSource:0}: Error finding container b181b772c5e3cd89a439d9a94f8d89464761441d95eca40fe7e72290a97290c0: Status 404 returned error can't find the container with id b181b772c5e3cd89a439d9a94f8d89464761441d95eca40fe7e72290a97290c0 Jul 24 18:19:29.709705 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:29.709670 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" event={"ID":"c8b758bb-0010-4f24-8040-132d81456f43","Type":"ContainerStarted","Data":"b181b772c5e3cd89a439d9a94f8d89464761441d95eca40fe7e72290a97290c0"} Jul 24 18:19:30.328235 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:19:30.328201 2559 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8b758bb_0010_4f24_8040_132d81456f43.slice/crio-conmon-b1dafd7d797e21d3411d154b7a94c2955f28a5d7354031c38b9271545abb1037.scope\": RecentStats: unable to find data in memory cache]" Jul 24 18:19:30.715376 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:30.715337 2559 generic.go:356] "Generic (PLEG): container finished" podID="c8b758bb-0010-4f24-8040-132d81456f43" containerID="b1dafd7d797e21d3411d154b7a94c2955f28a5d7354031c38b9271545abb1037" exitCode=0 Jul 24 18:19:30.715585 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:30.715390 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" event={"ID":"c8b758bb-0010-4f24-8040-132d81456f43","Type":"ContainerDied","Data":"b1dafd7d797e21d3411d154b7a94c2955f28a5d7354031c38b9271545abb1037"} Jul 24 18:19:31.852930 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:31.852904 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" Jul 24 18:19:31.910506 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:31.910459 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b758bb-0010-4f24-8040-132d81456f43-maas-api-service-ca\") pod \"c8b758bb-0010-4f24-8040-132d81456f43\" (UID: \"c8b758bb-0010-4f24-8040-132d81456f43\") " Jul 24 18:19:31.910893 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:31.910856 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8b758bb-0010-4f24-8040-132d81456f43-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8b758bb-0010-4f24-8040-132d81456f43" (UID: "c8b758bb-0010-4f24-8040-132d81456f43"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:19:32.011674 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:32.011582 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8b758bb-0010-4f24-8040-132d81456f43-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:19:32.725570 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:32.725525 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" event={"ID":"c8b758bb-0010-4f24-8040-132d81456f43","Type":"ContainerDied","Data":"b181b772c5e3cd89a439d9a94f8d89464761441d95eca40fe7e72290a97290c0"} Jul 24 18:19:32.725570 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:32.725569 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq" Jul 24 18:19:32.725821 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:32.725592 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b181b772c5e3cd89a439d9a94f8d89464761441d95eca40fe7e72290a97290c0" Jul 24 18:19:46.762152 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:46.762118 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:19:49.243663 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:49.243625 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq"] Jul 24 18:19:49.247749 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:49.247725 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ac2de9-wrtxq"] Jul 24 18:19:49.493338 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:49.493309 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8b758bb-0010-4f24-8040-132d81456f43" path="/var/lib/kubelet/pods/c8b758bb-0010-4f24-8040-132d81456f43/volumes" Jul 24 18:19:55.743918 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:55.743882 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f"] Jul 24 18:19:55.744345 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:19:55.744156 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" containerID="cri-o://4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1" gracePeriod=30 Jul 24 18:20:01.039839 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.039811 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:20:01.169026 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.168995 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-socket\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.169221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169046 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-token\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.169221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169075 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l78r7\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-kube-api-access-l78r7\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.169221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169119 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-certs\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.169221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169151 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a6078e-3aaa-4950-9032-3193b1ec10f5-istiod-ca-cert\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.169221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169191 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-envoy\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.169221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169215 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-podinfo\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.169528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169240 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-credential-socket\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.169528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169305 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:20:01.169528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169384 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:20:01.169697 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169648 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:20:01.169697 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169667 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39a6078e-3aaa-4950-9032-3193b1ec10f5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:20:01.169827 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.169773 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-data\") pod \"39a6078e-3aaa-4950-9032-3193b1ec10f5\" (UID: \"39a6078e-3aaa-4950-9032-3193b1ec10f5\") " Jul 24 18:20:01.170185 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.170125 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-credential-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.170185 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.170147 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.170185 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.170161 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-workload-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.170185 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.170169 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/39a6078e-3aaa-4950-9032-3193b1ec10f5-istiod-ca-cert\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.170395 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.170225 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-data" (OuterVolumeSpecName: "istio-data") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:20:01.171653 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.171631 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:20:01.171975 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.171945 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:20:01.171975 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.171959 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-kube-api-access-l78r7" (OuterVolumeSpecName: "kube-api-access-l78r7") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "kube-api-access-l78r7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:20:01.172078 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.172062 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-token" (OuterVolumeSpecName: "istio-token") pod "39a6078e-3aaa-4950-9032-3193b1ec10f5" (UID: "39a6078e-3aaa-4950-9032-3193b1ec10f5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:20:01.271301 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.271217 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-token\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.271301 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.271249 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l78r7\" (UniqueName: \"kubernetes.io/projected/39a6078e-3aaa-4950-9032-3193b1ec10f5-kube-api-access-l78r7\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.271301 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.271266 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-envoy\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.271301 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.271274 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-podinfo\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.271301 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.271282 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/39a6078e-3aaa-4950-9032-3193b1ec10f5-istio-data\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:20:01.860082 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.860050 2559 generic.go:356] "Generic (PLEG): container finished" podID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerID="4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1" exitCode=0 Jul 24 18:20:01.860287 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.860110 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" Jul 24 18:20:01.860287 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.860135 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" event={"ID":"39a6078e-3aaa-4950-9032-3193b1ec10f5","Type":"ContainerDied","Data":"4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1"} Jul 24 18:20:01.860287 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.860191 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f" event={"ID":"39a6078e-3aaa-4950-9032-3193b1ec10f5","Type":"ContainerDied","Data":"f3fa517b3b4b22717f0049824eaf11064caf269ccf905b9ca2cbba703f765373"} Jul 24 18:20:01.860287 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.860214 2559 scope.go:117] "RemoveContainer" containerID="4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1" Jul 24 18:20:01.869510 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.869301 2559 scope.go:117] "RemoveContainer" containerID="f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441" Jul 24 18:20:01.877346 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.877329 2559 scope.go:117] "RemoveContainer" containerID="4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1" Jul 24 18:20:01.877639 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:20:01.877618 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1\": container with ID starting with 4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1 not found: ID does not exist" containerID="4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1" Jul 24 18:20:01.877694 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.877651 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1"} err="failed to get container status \"4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1\": rpc error: code = NotFound desc = could not find container \"4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1\": container with ID starting with 4bc8e9723404053083a6721c38ab741b0d9e307b951752e8d9a5d4626e0d23b1 not found: ID does not exist" Jul 24 18:20:01.877694 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.877670 2559 scope.go:117] "RemoveContainer" containerID="f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441" Jul 24 18:20:01.877935 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:20:01.877917 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441\": container with ID starting with f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441 not found: ID does not exist" containerID="f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441" Jul 24 18:20:01.877993 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.877941 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441"} err="failed to get container status \"f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441\": rpc error: code = NotFound desc = could not find container \"f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441\": container with ID starting with f0b32b95eef3f558b433a12b40d58ab0bb8886c54306c8e86434c6a425749441 not found: ID does not exist" Jul 24 18:20:01.885893 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.885870 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f"] Jul 24 18:20:01.887422 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:01.887402 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-ac2de9-openshift-default-649f747fb6-ttj4f"] Jul 24 18:20:03.494102 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:03.494070 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" path="/var/lib/kubelet/pods/39a6078e-3aaa-4950-9032-3193b1ec10f5/volumes" Jul 24 18:20:03.951197 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:03.951163 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:20:11.960981 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:11.960948 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:20:21.755713 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.755634 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x"] Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.755970 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8b758bb-0010-4f24-8040-132d81456f43" containerName="revoke-keys" Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.755982 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8b758bb-0010-4f24-8040-132d81456f43" containerName="revoke-keys" Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.755994 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.755999 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.756008 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.756014 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.756066 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.756072 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="39a6078e-3aaa-4950-9032-3193b1ec10f5" containerName="istio-proxy" Jul 24 18:20:21.756110 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.756080 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8b758bb-0010-4f24-8040-132d81456f43" containerName="revoke-keys" Jul 24 18:20:21.760458 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.760442 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.763248 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.763228 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"kube-root-ca.crt\"" Jul 24 18:20:21.764285 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.764264 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"default-dockercfg-xlwnx\"" Jul 24 18:20:21.764285 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.764279 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"openshift-service-ca.crt\"" Jul 24 18:20:21.764435 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.764403 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"auth-test-model-2542df-kserve-self-signed-certs\"" Jul 24 18:20:21.775480 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.775458 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x"] Jul 24 18:20:21.852854 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.852817 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-home\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.852854 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.852859 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tmp-dir\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.853106 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.852875 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tls-certs\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.853106 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.852939 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-model-cache\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.853106 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.853014 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp4cj\" (UniqueName: \"kubernetes.io/projected/7c0eb967-6bf6-4497-abb1-cbed518d28aa-kube-api-access-vp4cj\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.853106 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.853060 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-dshm\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954277 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954245 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-home\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954277 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954293 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tmp-dir\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954549 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954317 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tls-certs\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954549 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954347 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-model-cache\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954549 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954400 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp4cj\" (UniqueName: \"kubernetes.io/projected/7c0eb967-6bf6-4497-abb1-cbed518d28aa-kube-api-access-vp4cj\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954549 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954440 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-dshm\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954760 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954711 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-home\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954760 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954732 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tmp-dir\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.954910 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.954774 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-model-cache\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.956909 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.956885 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-dshm\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.957256 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.957236 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tls-certs\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:21.963182 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:21.963157 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp4cj\" (UniqueName: \"kubernetes.io/projected/7c0eb967-6bf6-4497-abb1-cbed518d28aa-kube-api-access-vp4cj\") pod \"auth-test-model-2542df-kserve-66b75f8895-qjq4x\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:22.069298 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:22.069273 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:22.209238 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:22.209211 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x"] Jul 24 18:20:22.211564 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:20:22.211536 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c0eb967_6bf6_4497_abb1_cbed518d28aa.slice/crio-5a7eb32ea8c1f9a25a178e3876e503c5d82eab8199bc15fefa462c1f72d57b0a WatchSource:0}: Error finding container 5a7eb32ea8c1f9a25a178e3876e503c5d82eab8199bc15fefa462c1f72d57b0a: Status 404 returned error can't find the container with id 5a7eb32ea8c1f9a25a178e3876e503c5d82eab8199bc15fefa462c1f72d57b0a Jul 24 18:20:22.957391 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:22.957337 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" event={"ID":"7c0eb967-6bf6-4497-abb1-cbed518d28aa","Type":"ContainerStarted","Data":"89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035"} Jul 24 18:20:22.957906 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:22.957399 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" event={"ID":"7c0eb967-6bf6-4497-abb1-cbed518d28aa","Type":"ContainerStarted","Data":"5a7eb32ea8c1f9a25a178e3876e503c5d82eab8199bc15fefa462c1f72d57b0a"} Jul 24 18:20:22.975959 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:22.975911 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" podStartSLOduration=1.975893452 podStartE2EDuration="1.975893452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:20:22.975000723 +0000 UTC m=+2248.014794763" watchObservedRunningTime="2026-07-24 18:20:22.975893452 +0000 UTC m=+2248.015687493" Jul 24 18:20:24.457409 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:24.457365 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:20:32.070381 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:32.070336 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:32.070845 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:32.070395 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:32.082683 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:32.082659 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:33.025861 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:33.025823 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:20:55.250533 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.250492 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8"] Jul 24 18:20:55.253865 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.253846 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.256840 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.256818 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"openshift-service-ca.crt\"" Jul 24 18:20:55.257894 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.257876 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"default-dockercfg-h4qkl\"" Jul 24 18:20:55.257894 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.257884 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"kube-root-ca.crt\"" Jul 24 18:20:55.258047 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.257879 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"auth-test-model-f556c1-kserve-self-signed-certs\"" Jul 24 18:20:55.263960 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.263936 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8"] Jul 24 18:20:55.350588 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.350551 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg5rf\" (UniqueName: \"kubernetes.io/projected/5dbfb975-859e-42e7-9943-0bbacae29573-kube-api-access-sg5rf\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.350842 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.350600 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbfb975-859e-42e7-9943-0bbacae29573-tls-certs\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.350842 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.350623 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-model-cache\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.350842 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.350716 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-dshm\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.350842 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.350753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-home\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.350842 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.350777 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-tmp-dir\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.452366 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.452325 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg5rf\" (UniqueName: \"kubernetes.io/projected/5dbfb975-859e-42e7-9943-0bbacae29573-kube-api-access-sg5rf\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.452543 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.452408 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbfb975-859e-42e7-9943-0bbacae29573-tls-certs\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.452543 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.452447 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-model-cache\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.452543 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.452500 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-dshm\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.452543 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.452527 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-home\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.452744 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.452551 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-tmp-dir\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.453054 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.453002 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-model-cache\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.453171 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.453076 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-tmp-dir\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.453171 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.453100 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-home\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.455025 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.455005 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-dshm\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.455989 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.455971 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"auth-test-model-f556c1-kserve-self-signed-certs\"" Jul 24 18:20:55.463478 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.463456 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"kube-root-ca.crt\"" Jul 24 18:20:55.465040 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.465025 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbfb975-859e-42e7-9943-0bbacae29573-tls-certs\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.473821 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.473801 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"openshift-service-ca.crt\"" Jul 24 18:20:55.483447 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.483429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg5rf\" (UniqueName: \"kubernetes.io/projected/5dbfb975-859e-42e7-9943-0bbacae29573-kube-api-access-sg5rf\") pod \"auth-test-model-f556c1-kserve-6f54659b67-nlnd8\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.567295 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.567270 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"default-dockercfg-h4qkl\"" Jul 24 18:20:55.574946 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.574931 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:20:55.709148 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.709119 2559 scope.go:117] "RemoveContainer" containerID="74db8b0e250e90c5544f41a2b674a4e70f0f0985f4378804055ed7cb4ee22bd2" Jul 24 18:20:55.710721 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:55.710698 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8"] Jul 24 18:20:55.713936 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:20:55.713896 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5dbfb975_859e_42e7_9943_0bbacae29573.slice/crio-ac8917b63d0400443be4cff186ab329dd9d83469be82bd8c3da497bcee6ebc32 WatchSource:0}: Error finding container ac8917b63d0400443be4cff186ab329dd9d83469be82bd8c3da497bcee6ebc32: Status 404 returned error can't find the container with id ac8917b63d0400443be4cff186ab329dd9d83469be82bd8c3da497bcee6ebc32 Jul 24 18:20:56.124773 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:56.124735 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" event={"ID":"5dbfb975-859e-42e7-9943-0bbacae29573","Type":"ContainerStarted","Data":"0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8"} Jul 24 18:20:56.124773 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:56.124778 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" event={"ID":"5dbfb975-859e-42e7-9943-0bbacae29573","Type":"ContainerStarted","Data":"ac8917b63d0400443be4cff186ab329dd9d83469be82bd8c3da497bcee6ebc32"} Jul 24 18:20:56.143017 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:56.142955 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" podStartSLOduration=1.142935026 podStartE2EDuration="1.142935026s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:20:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:20:56.142476611 +0000 UTC m=+2281.182270653" watchObservedRunningTime="2026-07-24 18:20:56.142935026 +0000 UTC m=+2281.182729045" Jul 24 18:20:57.846633 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:20:57.846599 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:21:05.575479 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:05.575443 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:21:05.576028 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:05.575515 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:21:05.587602 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:05.587577 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:21:06.181442 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:06.181407 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:21:15.449869 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:15.449832 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:21:26.456383 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:26.456344 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:21:37.548910 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:37.548815 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:21:46.861656 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:46.861547 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:21:59.253369 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:21:59.253328 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:22:11.055666 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:11.055634 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:22:22.050915 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:22.050879 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:22:29.737954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:29.737915 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x"] Jul 24 18:22:30.653017 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:30.652981 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8"] Jul 24 18:22:32.856501 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:32.856469 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:22:44.758490 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:44.758437 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" podUID="7c0eb967-6bf6-4497-abb1-cbed518d28aa" containerName="main" containerID="cri-o://89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035" gracePeriod=45 Jul 24 18:22:45.017364 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.017294 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:22:45.129063 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129030 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-home\") pod \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " Jul 24 18:22:45.129063 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129067 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp4cj\" (UniqueName: \"kubernetes.io/projected/7c0eb967-6bf6-4497-abb1-cbed518d28aa-kube-api-access-vp4cj\") pod \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " Jul 24 18:22:45.129324 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129119 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tls-certs\") pod \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " Jul 24 18:22:45.129324 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129165 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-model-cache\") pod \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " Jul 24 18:22:45.129324 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129183 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-dshm\") pod \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " Jul 24 18:22:45.129324 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129236 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tmp-dir\") pod \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\" (UID: \"7c0eb967-6bf6-4497-abb1-cbed518d28aa\") " Jul 24 18:22:45.129553 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129371 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-home" (OuterVolumeSpecName: "home") pod "7c0eb967-6bf6-4497-abb1-cbed518d28aa" (UID: "7c0eb967-6bf6-4497-abb1-cbed518d28aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:22:45.129553 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129489 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "7c0eb967-6bf6-4497-abb1-cbed518d28aa" (UID: "7c0eb967-6bf6-4497-abb1-cbed518d28aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:22:45.129553 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129516 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:45.129668 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.129528 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7c0eb967-6bf6-4497-abb1-cbed518d28aa" (UID: "7c0eb967-6bf6-4497-abb1-cbed518d28aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:22:45.131600 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.131567 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7c0eb967-6bf6-4497-abb1-cbed518d28aa" (UID: "7c0eb967-6bf6-4497-abb1-cbed518d28aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:22:45.131718 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.131598 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-dshm" (OuterVolumeSpecName: "dshm") pod "7c0eb967-6bf6-4497-abb1-cbed518d28aa" (UID: "7c0eb967-6bf6-4497-abb1-cbed518d28aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:22:45.131718 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.131611 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7c0eb967-6bf6-4497-abb1-cbed518d28aa-kube-api-access-vp4cj" (OuterVolumeSpecName: "kube-api-access-vp4cj") pod "7c0eb967-6bf6-4497-abb1-cbed518d28aa" (UID: "7c0eb967-6bf6-4497-abb1-cbed518d28aa"). InnerVolumeSpecName "kube-api-access-vp4cj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:22:45.230265 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.230229 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:45.230265 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.230259 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:45.230265 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.230267 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:45.230509 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.230278 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp4cj\" (UniqueName: \"kubernetes.io/projected/7c0eb967-6bf6-4497-abb1-cbed518d28aa-kube-api-access-vp4cj\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:45.230509 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.230286 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7c0eb967-6bf6-4497-abb1-cbed518d28aa-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:45.631045 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.631012 2559 generic.go:356] "Generic (PLEG): container finished" podID="7c0eb967-6bf6-4497-abb1-cbed518d28aa" containerID="89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035" exitCode=0 Jul 24 18:22:45.631228 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.631051 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" event={"ID":"7c0eb967-6bf6-4497-abb1-cbed518d28aa","Type":"ContainerDied","Data":"89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035"} Jul 24 18:22:45.631228 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.631056 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" Jul 24 18:22:45.631228 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.631082 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x" event={"ID":"7c0eb967-6bf6-4497-abb1-cbed518d28aa","Type":"ContainerDied","Data":"5a7eb32ea8c1f9a25a178e3876e503c5d82eab8199bc15fefa462c1f72d57b0a"} Jul 24 18:22:45.631228 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.631099 2559 scope.go:117] "RemoveContainer" containerID="89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035" Jul 24 18:22:45.643172 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.643153 2559 scope.go:117] "RemoveContainer" containerID="89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035" Jul 24 18:22:45.643539 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:22:45.643517 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035\": container with ID starting with 89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035 not found: ID does not exist" containerID="89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035" Jul 24 18:22:45.643619 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.643551 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035"} err="failed to get container status \"89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035\": rpc error: code = NotFound desc = could not find container \"89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035\": container with ID starting with 89191a2b961b4fe9e4925a571e4ea85fec3562e93a1a1ea638d6590076871035 not found: ID does not exist" Jul 24 18:22:45.650119 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.650093 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x"] Jul 24 18:22:45.653606 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.653579 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/auth-test-model-2542df-kserve-66b75f8895-qjq4x"] Jul 24 18:22:45.672883 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.672842 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" podUID="5dbfb975-859e-42e7-9943-0bbacae29573" containerName="main" containerID="cri-o://0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8" gracePeriod=45 Jul 24 18:22:45.931193 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:45.931167 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:22:46.038701 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.038604 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-dshm\") pod \"5dbfb975-859e-42e7-9943-0bbacae29573\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " Jul 24 18:22:46.038701 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.038660 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg5rf\" (UniqueName: \"kubernetes.io/projected/5dbfb975-859e-42e7-9943-0bbacae29573-kube-api-access-sg5rf\") pod \"5dbfb975-859e-42e7-9943-0bbacae29573\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " Jul 24 18:22:46.038701 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.038688 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-model-cache\") pod \"5dbfb975-859e-42e7-9943-0bbacae29573\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " Jul 24 18:22:46.039027 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.038709 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-home\") pod \"5dbfb975-859e-42e7-9943-0bbacae29573\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " Jul 24 18:22:46.039027 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.038724 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-tmp-dir\") pod \"5dbfb975-859e-42e7-9943-0bbacae29573\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " Jul 24 18:22:46.039027 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.038761 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbfb975-859e-42e7-9943-0bbacae29573-tls-certs\") pod \"5dbfb975-859e-42e7-9943-0bbacae29573\" (UID: \"5dbfb975-859e-42e7-9943-0bbacae29573\") " Jul 24 18:22:46.039027 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.039007 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-home" (OuterVolumeSpecName: "home") pod "5dbfb975-859e-42e7-9943-0bbacae29573" (UID: "5dbfb975-859e-42e7-9943-0bbacae29573"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:22:46.039192 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.039026 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5dbfb975-859e-42e7-9943-0bbacae29573" (UID: "5dbfb975-859e-42e7-9943-0bbacae29573"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:22:46.039192 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.039111 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-model-cache" (OuterVolumeSpecName: "model-cache") pod "5dbfb975-859e-42e7-9943-0bbacae29573" (UID: "5dbfb975-859e-42e7-9943-0bbacae29573"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:22:46.041197 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.041165 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dbfb975-859e-42e7-9943-0bbacae29573-kube-api-access-sg5rf" (OuterVolumeSpecName: "kube-api-access-sg5rf") pod "5dbfb975-859e-42e7-9943-0bbacae29573" (UID: "5dbfb975-859e-42e7-9943-0bbacae29573"). InnerVolumeSpecName "kube-api-access-sg5rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:22:46.041197 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.041189 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-dshm" (OuterVolumeSpecName: "dshm") pod "5dbfb975-859e-42e7-9943-0bbacae29573" (UID: "5dbfb975-859e-42e7-9943-0bbacae29573"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:22:46.041386 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.041231 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5dbfb975-859e-42e7-9943-0bbacae29573-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5dbfb975-859e-42e7-9943-0bbacae29573" (UID: "5dbfb975-859e-42e7-9943-0bbacae29573"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:22:46.140020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.139975 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg5rf\" (UniqueName: \"kubernetes.io/projected/5dbfb975-859e-42e7-9943-0bbacae29573-kube-api-access-sg5rf\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:46.140020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.140011 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:46.140020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.140023 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:46.140279 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.140050 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:46.140279 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.140059 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5dbfb975-859e-42e7-9943-0bbacae29573-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:46.140279 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.140066 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5dbfb975-859e-42e7-9943-0bbacae29573-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:22:46.637525 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.637491 2559 generic.go:356] "Generic (PLEG): container finished" podID="5dbfb975-859e-42e7-9943-0bbacae29573" containerID="0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8" exitCode=0 Jul 24 18:22:46.637717 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.637532 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" event={"ID":"5dbfb975-859e-42e7-9943-0bbacae29573","Type":"ContainerDied","Data":"0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8"} Jul 24 18:22:46.637717 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.637546 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" Jul 24 18:22:46.637717 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.637578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8" event={"ID":"5dbfb975-859e-42e7-9943-0bbacae29573","Type":"ContainerDied","Data":"ac8917b63d0400443be4cff186ab329dd9d83469be82bd8c3da497bcee6ebc32"} Jul 24 18:22:46.637717 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.637597 2559 scope.go:117] "RemoveContainer" containerID="0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8" Jul 24 18:22:46.646835 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.646806 2559 scope.go:117] "RemoveContainer" containerID="0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8" Jul 24 18:22:46.647104 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:22:46.647084 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8\": container with ID starting with 0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8 not found: ID does not exist" containerID="0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8" Jul 24 18:22:46.647174 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.647114 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8"} err="failed to get container status \"0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8\": rpc error: code = NotFound desc = could not find container \"0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8\": container with ID starting with 0f9f5bbd3dd9766d071190b40a17ccbb8c22f68ab92c5ed4aec6959a49984cd8 not found: ID does not exist" Jul 24 18:22:46.664592 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.664558 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8"] Jul 24 18:22:46.669904 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:46.669877 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/auth-test-model-f556c1-kserve-6f54659b67-nlnd8"] Jul 24 18:22:47.494115 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:47.494085 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dbfb975-859e-42e7-9943-0bbacae29573" path="/var/lib/kubelet/pods/5dbfb975-859e-42e7-9943-0bbacae29573/volumes" Jul 24 18:22:47.494503 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:47.494472 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c0eb967-6bf6-4497-abb1-cbed518d28aa" path="/var/lib/kubelet/pods/7c0eb967-6bf6-4497-abb1-cbed518d28aa/volumes" Jul 24 18:22:55.563489 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.563452 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp"] Jul 24 18:22:55.563895 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.563878 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c0eb967-6bf6-4497-abb1-cbed518d28aa" containerName="main" Jul 24 18:22:55.563942 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.563897 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c0eb967-6bf6-4497-abb1-cbed518d28aa" containerName="main" Jul 24 18:22:55.563942 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.563910 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dbfb975-859e-42e7-9943-0bbacae29573" containerName="main" Jul 24 18:22:55.563942 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.563915 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dbfb975-859e-42e7-9943-0bbacae29573" containerName="main" Jul 24 18:22:55.564036 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.563995 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dbfb975-859e-42e7-9943-0bbacae29573" containerName="main" Jul 24 18:22:55.564036 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.564006 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c0eb967-6bf6-4497-abb1-cbed518d28aa" containerName="main" Jul 24 18:22:55.568406 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.568390 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.571206 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.571186 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"kube-root-ca.crt\"" Jul 24 18:22:55.572425 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.572406 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"openshift-service-ca.crt\"" Jul 24 18:22:55.572527 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.572427 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"default-dockercfg-xlwnx\"" Jul 24 18:22:55.572527 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.572483 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"sub-test-model-2542df-kserve-self-signed-certs\"" Jul 24 18:22:55.577770 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.577749 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp"] Jul 24 18:22:55.613748 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.613725 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:22:55.617755 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.617732 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:22:55.622482 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.622454 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-home\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.622623 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.622501 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgnd\" (UniqueName: \"kubernetes.io/projected/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-kube-api-access-twgnd\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.622623 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.622608 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-model-cache\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.622742 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.622641 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tls-certs\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.622831 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.622779 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-dshm\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.622890 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.622843 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tmp-dir\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723257 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723228 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-dshm\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723257 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723259 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tmp-dir\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723496 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723281 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-home\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723496 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723306 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgnd\" (UniqueName: \"kubernetes.io/projected/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-kube-api-access-twgnd\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723496 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723333 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-model-cache\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723496 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723357 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tls-certs\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723726 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723707 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tmp-dir\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723775 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723761 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-home\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.723845 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.723821 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-model-cache\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.725696 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.725674 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-dshm\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.725867 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.725848 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tls-certs\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.744303 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.744279 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgnd\" (UniqueName: \"kubernetes.io/projected/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-kube-api-access-twgnd\") pod \"sub-test-model-2542df-kserve-f4d5c8497-8cvpp\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:55.878185 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:55.878098 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:22:56.021157 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:56.021133 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp"] Jul 24 18:22:56.023297 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:22:56.023274 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88c64bf8_a55f_4abc_ab02_ad9926bf8cec.slice/crio-00cf082297d1208da257f7aa83557dfee88fb7f4d983355b0737e145d643a31f WatchSource:0}: Error finding container 00cf082297d1208da257f7aa83557dfee88fb7f4d983355b0737e145d643a31f: Status 404 returned error can't find the container with id 00cf082297d1208da257f7aa83557dfee88fb7f4d983355b0737e145d643a31f Jul 24 18:22:56.685575 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:56.685524 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" event={"ID":"88c64bf8-a55f-4abc-ab02-ad9926bf8cec","Type":"ContainerStarted","Data":"5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894"} Jul 24 18:22:56.685575 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:56.685578 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" event={"ID":"88c64bf8-a55f-4abc-ab02-ad9926bf8cec","Type":"ContainerStarted","Data":"00cf082297d1208da257f7aa83557dfee88fb7f4d983355b0737e145d643a31f"} Jul 24 18:22:56.707169 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:56.707114 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" podStartSLOduration=1.707098562 podStartE2EDuration="1.707098562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:22:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:22:56.704503531 +0000 UTC m=+2401.744297572" watchObservedRunningTime="2026-07-24 18:22:56.707098562 +0000 UTC m=+2401.746892602" Jul 24 18:22:58.254599 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:22:58.254566 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:23:05.879226 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:05.879183 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:23:05.879651 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:05.879275 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:23:05.891935 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:05.891909 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:23:06.744569 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:06.744531 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:23:22.430694 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.430607 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg"] Jul 24 18:23:22.434232 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.434210 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.438063 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.438036 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"sub-test-model-f556c1-kserve-self-signed-certs\"" Jul 24 18:23:22.438063 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.438051 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"default-dockercfg-h4qkl\"" Jul 24 18:23:22.438263 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.438073 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"openshift-service-ca.crt\"" Jul 24 18:23:22.438263 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.438085 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"kube-root-ca.crt\"" Jul 24 18:23:22.443205 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.443177 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg"] Jul 24 18:23:22.570261 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.570224 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-model-cache\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.570490 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.570279 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-home\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.570490 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.570301 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-tmp-dir\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.570490 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.570331 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-dshm\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.570490 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.570357 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqpmp\" (UniqueName: \"kubernetes.io/projected/7bd221dd-bf8e-4763-881f-70dc93d7233d-kube-api-access-dqpmp\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.570490 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.570396 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd221dd-bf8e-4763-881f-70dc93d7233d-tls-certs\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.671427 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671387 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-model-cache\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.671427 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-home\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.671689 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671460 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-tmp-dir\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.671689 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671481 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-dshm\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.671689 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671507 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqpmp\" (UniqueName: \"kubernetes.io/projected/7bd221dd-bf8e-4763-881f-70dc93d7233d-kube-api-access-dqpmp\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.671689 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671531 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd221dd-bf8e-4763-881f-70dc93d7233d-tls-certs\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.671954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671873 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-home\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.671954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671929 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-model-cache\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.672038 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.671990 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-tmp-dir\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.674013 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.673996 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-dshm\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.674193 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.674177 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd221dd-bf8e-4763-881f-70dc93d7233d-tls-certs\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.680637 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.680611 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqpmp\" (UniqueName: \"kubernetes.io/projected/7bd221dd-bf8e-4763-881f-70dc93d7233d-kube-api-access-dqpmp\") pod \"sub-test-model-f556c1-kserve-746fcf7b59-tnxpg\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.744579 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.744471 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:22.890946 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:22.890918 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg"] Jul 24 18:23:22.892694 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:23:22.892658 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd221dd_bf8e_4763_881f_70dc93d7233d.slice/crio-5e49d0a33e19b7a854ad5ff4b05a2f1d30cd4727b1f887631489a28529b38f67 WatchSource:0}: Error finding container 5e49d0a33e19b7a854ad5ff4b05a2f1d30cd4727b1f887631489a28529b38f67: Status 404 returned error can't find the container with id 5e49d0a33e19b7a854ad5ff4b05a2f1d30cd4727b1f887631489a28529b38f67 Jul 24 18:23:23.822686 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:23.822653 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" event={"ID":"7bd221dd-bf8e-4763-881f-70dc93d7233d","Type":"ContainerStarted","Data":"d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484"} Jul 24 18:23:23.822686 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:23.822691 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" event={"ID":"7bd221dd-bf8e-4763-881f-70dc93d7233d","Type":"ContainerStarted","Data":"5e49d0a33e19b7a854ad5ff4b05a2f1d30cd4727b1f887631489a28529b38f67"} Jul 24 18:23:23.844390 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:23.844340 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" podStartSLOduration=1.844324904 podStartE2EDuration="1.844324904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:23:23.843348497 +0000 UTC m=+2428.883142541" watchObservedRunningTime="2026-07-24 18:23:23.844324904 +0000 UTC m=+2428.884118945" Jul 24 18:23:24.852436 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:24.852400 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:23:32.745073 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:32.745016 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:32.745073 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:32.745084 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:32.757547 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:32.757511 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:32.876622 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:32.876595 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:23:49.254598 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:49.254560 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:23:57.963730 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:23:57.963693 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:24:03.723442 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:03.723409 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp"] Jul 24 18:24:04.619668 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:04.619628 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg"] Jul 24 18:24:09.860483 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:09.860448 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:24:11.056975 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.056947 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch"] Jul 24 18:24:11.060305 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.060285 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.063292 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.063268 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-2542df\"/\"rate-test-model-2542df-kserve-self-signed-certs\"" Jul 24 18:24:11.074485 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.074456 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch"] Jul 24 18:24:11.201199 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.201162 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-tmp-dir\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.201402 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.201218 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sw2n\" (UniqueName: \"kubernetes.io/projected/e42143eb-1659-46cb-9c51-47de4a2a96f7-kube-api-access-2sw2n\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.201402 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.201241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-model-cache\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.201402 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.201267 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42143eb-1659-46cb-9c51-47de4a2a96f7-tls-certs\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.201402 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.201368 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-dshm\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.201636 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.201417 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-home\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302173 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302138 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-dshm\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302173 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302176 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-home\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302436 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302256 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-tmp-dir\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302436 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302277 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sw2n\" (UniqueName: \"kubernetes.io/projected/e42143eb-1659-46cb-9c51-47de4a2a96f7-kube-api-access-2sw2n\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302436 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302301 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-model-cache\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302436 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302344 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42143eb-1659-46cb-9c51-47de4a2a96f7-tls-certs\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302688 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302651 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-home\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302835 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302690 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-tmp-dir\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.302835 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.302727 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-model-cache\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.304499 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.304479 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-dshm\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.304848 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.304831 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42143eb-1659-46cb-9c51-47de4a2a96f7-tls-certs\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.310498 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.310429 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sw2n\" (UniqueName: \"kubernetes.io/projected/e42143eb-1659-46cb-9c51-47de4a2a96f7-kube-api-access-2sw2n\") pod \"rate-test-model-2542df-kserve-db7667679-cvwch\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.372848 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.372816 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:11.511384 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:11.511359 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch"] Jul 24 18:24:11.513431 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:24:11.513402 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode42143eb_1659_46cb_9c51_47de4a2a96f7.slice/crio-0faea21ef5ae146c7c254a008a3e4b2cc18db7a71bed3d9d049a602e127586b6 WatchSource:0}: Error finding container 0faea21ef5ae146c7c254a008a3e4b2cc18db7a71bed3d9d049a602e127586b6: Status 404 returned error can't find the container with id 0faea21ef5ae146c7c254a008a3e4b2cc18db7a71bed3d9d049a602e127586b6 Jul 24 18:24:12.058714 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:12.058628 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" event={"ID":"e42143eb-1659-46cb-9c51-47de4a2a96f7","Type":"ContainerStarted","Data":"54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a"} Jul 24 18:24:12.058714 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:12.058669 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" event={"ID":"e42143eb-1659-46cb-9c51-47de4a2a96f7","Type":"ContainerStarted","Data":"0faea21ef5ae146c7c254a008a3e4b2cc18db7a71bed3d9d049a602e127586b6"} Jul 24 18:24:12.077209 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:12.077162 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" podStartSLOduration=1.077145869 podStartE2EDuration="1.077145869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:24:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:24:12.075607025 +0000 UTC m=+2477.115401067" watchObservedRunningTime="2026-07-24 18:24:12.077145869 +0000 UTC m=+2477.116939911" Jul 24 18:24:17.254634 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:17.254591 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:24:18.745529 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:18.745477 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" podUID="88c64bf8-a55f-4abc-ab02-ad9926bf8cec" containerName="main" containerID="cri-o://5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894" gracePeriod=45 Jul 24 18:24:19.002298 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.002231 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:24:19.092803 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.092754 2559 generic.go:356] "Generic (PLEG): container finished" podID="88c64bf8-a55f-4abc-ab02-ad9926bf8cec" containerID="5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894" exitCode=0 Jul 24 18:24:19.092968 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.092814 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" Jul 24 18:24:19.092968 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.092838 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" event={"ID":"88c64bf8-a55f-4abc-ab02-ad9926bf8cec","Type":"ContainerDied","Data":"5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894"} Jul 24 18:24:19.092968 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.092886 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp" event={"ID":"88c64bf8-a55f-4abc-ab02-ad9926bf8cec","Type":"ContainerDied","Data":"00cf082297d1208da257f7aa83557dfee88fb7f4d983355b0737e145d643a31f"} Jul 24 18:24:19.092968 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.092910 2559 scope.go:117] "RemoveContainer" containerID="5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894" Jul 24 18:24:19.101401 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.101381 2559 scope.go:117] "RemoveContainer" containerID="5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894" Jul 24 18:24:19.101679 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:24:19.101660 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894\": container with ID starting with 5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894 not found: ID does not exist" containerID="5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894" Jul 24 18:24:19.101754 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.101693 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894"} err="failed to get container status \"5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894\": rpc error: code = NotFound desc = could not find container \"5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894\": container with ID starting with 5dc35833628aaa94ee9a4835ae6a7ad077d4295901b0eec37f7aebcdbfff2894 not found: ID does not exist" Jul 24 18:24:19.176678 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.176643 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-home\") pod \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " Jul 24 18:24:19.176678 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.176678 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twgnd\" (UniqueName: \"kubernetes.io/projected/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-kube-api-access-twgnd\") pod \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " Jul 24 18:24:19.176909 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.176697 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tmp-dir\") pod \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " Jul 24 18:24:19.176909 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.176780 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-model-cache\") pod \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " Jul 24 18:24:19.176909 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.176830 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tls-certs\") pod \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " Jul 24 18:24:19.176909 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.176854 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-dshm\") pod \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\" (UID: \"88c64bf8-a55f-4abc-ab02-ad9926bf8cec\") " Jul 24 18:24:19.177115 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.176964 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-home" (OuterVolumeSpecName: "home") pod "88c64bf8-a55f-4abc-ab02-ad9926bf8cec" (UID: "88c64bf8-a55f-4abc-ab02-ad9926bf8cec"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:24:19.177115 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.176982 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88c64bf8-a55f-4abc-ab02-ad9926bf8cec" (UID: "88c64bf8-a55f-4abc-ab02-ad9926bf8cec"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:24:19.177226 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.177123 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:19.177226 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.177143 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:19.177226 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.177123 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-model-cache" (OuterVolumeSpecName: "model-cache") pod "88c64bf8-a55f-4abc-ab02-ad9926bf8cec" (UID: "88c64bf8-a55f-4abc-ab02-ad9926bf8cec"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:24:19.179025 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.178995 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-kube-api-access-twgnd" (OuterVolumeSpecName: "kube-api-access-twgnd") pod "88c64bf8-a55f-4abc-ab02-ad9926bf8cec" (UID: "88c64bf8-a55f-4abc-ab02-ad9926bf8cec"). InnerVolumeSpecName "kube-api-access-twgnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:24:19.179136 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.179043 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-dshm" (OuterVolumeSpecName: "dshm") pod "88c64bf8-a55f-4abc-ab02-ad9926bf8cec" (UID: "88c64bf8-a55f-4abc-ab02-ad9926bf8cec"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:24:19.179136 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.179064 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88c64bf8-a55f-4abc-ab02-ad9926bf8cec" (UID: "88c64bf8-a55f-4abc-ab02-ad9926bf8cec"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:24:19.278270 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.278181 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:19.278270 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.278213 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgnd\" (UniqueName: \"kubernetes.io/projected/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-kube-api-access-twgnd\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:19.278270 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.278224 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:19.278270 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.278233 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88c64bf8-a55f-4abc-ab02-ad9926bf8cec-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:19.417060 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.417029 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp"] Jul 24 18:24:19.421283 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.421260 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/sub-test-model-2542df-kserve-f4d5c8497-8cvpp"] Jul 24 18:24:19.493403 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.493369 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88c64bf8-a55f-4abc-ab02-ad9926bf8cec" path="/var/lib/kubelet/pods/88c64bf8-a55f-4abc-ab02-ad9926bf8cec/volumes" Jul 24 18:24:19.638849 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.638801 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" podUID="7bd221dd-bf8e-4763-881f-70dc93d7233d" containerName="main" containerID="cri-o://d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484" gracePeriod=45 Jul 24 18:24:19.904077 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:19.904053 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:24:20.086252 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086212 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-model-cache\") pod \"7bd221dd-bf8e-4763-881f-70dc93d7233d\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " Jul 24 18:24:20.086252 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086263 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-home\") pod \"7bd221dd-bf8e-4763-881f-70dc93d7233d\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " Jul 24 18:24:20.086493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086305 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-tmp-dir\") pod \"7bd221dd-bf8e-4763-881f-70dc93d7233d\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " Jul 24 18:24:20.086493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086345 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-dshm\") pod \"7bd221dd-bf8e-4763-881f-70dc93d7233d\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " Jul 24 18:24:20.086493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086373 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd221dd-bf8e-4763-881f-70dc93d7233d-tls-certs\") pod \"7bd221dd-bf8e-4763-881f-70dc93d7233d\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " Jul 24 18:24:20.086493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086412 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqpmp\" (UniqueName: \"kubernetes.io/projected/7bd221dd-bf8e-4763-881f-70dc93d7233d-kube-api-access-dqpmp\") pod \"7bd221dd-bf8e-4763-881f-70dc93d7233d\" (UID: \"7bd221dd-bf8e-4763-881f-70dc93d7233d\") " Jul 24 18:24:20.086705 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086551 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-home" (OuterVolumeSpecName: "home") pod "7bd221dd-bf8e-4763-881f-70dc93d7233d" (UID: "7bd221dd-bf8e-4763-881f-70dc93d7233d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:24:20.086705 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086557 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-model-cache" (OuterVolumeSpecName: "model-cache") pod "7bd221dd-bf8e-4763-881f-70dc93d7233d" (UID: "7bd221dd-bf8e-4763-881f-70dc93d7233d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:24:20.086705 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.086605 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7bd221dd-bf8e-4763-881f-70dc93d7233d" (UID: "7bd221dd-bf8e-4763-881f-70dc93d7233d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:24:20.088690 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.088661 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bd221dd-bf8e-4763-881f-70dc93d7233d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7bd221dd-bf8e-4763-881f-70dc93d7233d" (UID: "7bd221dd-bf8e-4763-881f-70dc93d7233d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:24:20.088858 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.088708 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-dshm" (OuterVolumeSpecName: "dshm") pod "7bd221dd-bf8e-4763-881f-70dc93d7233d" (UID: "7bd221dd-bf8e-4763-881f-70dc93d7233d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:24:20.088858 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.088714 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd221dd-bf8e-4763-881f-70dc93d7233d-kube-api-access-dqpmp" (OuterVolumeSpecName: "kube-api-access-dqpmp") pod "7bd221dd-bf8e-4763-881f-70dc93d7233d" (UID: "7bd221dd-bf8e-4763-881f-70dc93d7233d"). InnerVolumeSpecName "kube-api-access-dqpmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:24:20.099505 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.099477 2559 generic.go:356] "Generic (PLEG): container finished" podID="7bd221dd-bf8e-4763-881f-70dc93d7233d" containerID="d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484" exitCode=0 Jul 24 18:24:20.099650 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.099514 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" Jul 24 18:24:20.099650 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.099563 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" event={"ID":"7bd221dd-bf8e-4763-881f-70dc93d7233d","Type":"ContainerDied","Data":"d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484"} Jul 24 18:24:20.099650 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.099622 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg" event={"ID":"7bd221dd-bf8e-4763-881f-70dc93d7233d","Type":"ContainerDied","Data":"5e49d0a33e19b7a854ad5ff4b05a2f1d30cd4727b1f887631489a28529b38f67"} Jul 24 18:24:20.099650 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.099643 2559 scope.go:117] "RemoveContainer" containerID="d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484" Jul 24 18:24:20.108628 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.108609 2559 scope.go:117] "RemoveContainer" containerID="d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484" Jul 24 18:24:20.108939 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:24:20.108921 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484\": container with ID starting with d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484 not found: ID does not exist" containerID="d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484" Jul 24 18:24:20.109027 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.108962 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484"} err="failed to get container status \"d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484\": rpc error: code = NotFound desc = could not find container \"d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484\": container with ID starting with d2ef732121b6a8d922b53737fcffdefd2af9d71edfc60e50e810acbe77d31484 not found: ID does not exist" Jul 24 18:24:20.120760 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.120736 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg"] Jul 24 18:24:20.124825 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.124805 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/sub-test-model-f556c1-kserve-746fcf7b59-tnxpg"] Jul 24 18:24:20.187929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.187849 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:20.187929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.187876 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:20.187929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.187884 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bd221dd-bf8e-4763-881f-70dc93d7233d-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:20.187929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.187892 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqpmp\" (UniqueName: \"kubernetes.io/projected/7bd221dd-bf8e-4763-881f-70dc93d7233d-kube-api-access-dqpmp\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:20.187929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.187902 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:20.187929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:20.187911 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bd221dd-bf8e-4763-881f-70dc93d7233d-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:24:21.373183 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:21.373143 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:21.373594 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:21.373207 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:21.385674 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:21.385649 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:21.493701 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:21.493671 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bd221dd-bf8e-4763-881f-70dc93d7233d" path="/var/lib/kubelet/pods/7bd221dd-bf8e-4763-881f-70dc93d7233d/volumes" Jul 24 18:24:22.121634 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:22.121608 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:24:31.256774 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.256725 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6"] Jul 24 18:24:31.257266 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.257120 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bd221dd-bf8e-4763-881f-70dc93d7233d" containerName="main" Jul 24 18:24:31.257266 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.257133 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bd221dd-bf8e-4763-881f-70dc93d7233d" containerName="main" Jul 24 18:24:31.257266 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.257158 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88c64bf8-a55f-4abc-ab02-ad9926bf8cec" containerName="main" Jul 24 18:24:31.257266 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.257164 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="88c64bf8-a55f-4abc-ab02-ad9926bf8cec" containerName="main" Jul 24 18:24:31.257266 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.257227 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="88c64bf8-a55f-4abc-ab02-ad9926bf8cec" containerName="main" Jul 24 18:24:31.257266 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.257239 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bd221dd-bf8e-4763-881f-70dc93d7233d" containerName="main" Jul 24 18:24:31.261778 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.261759 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.265001 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.264967 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"kube-root-ca.crt\"" Jul 24 18:24:31.265001 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.264967 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"openshift-service-ca.crt\"" Jul 24 18:24:31.266175 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.266157 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"default-dockercfg-h4qkl\"" Jul 24 18:24:31.266309 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.266213 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f556c1\"/\"rate-test-model-f556c1-kserve-self-signed-certs\"" Jul 24 18:24:31.271635 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.271611 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6"] Jul 24 18:24:31.287491 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.287448 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0444d15e-1c92-4f39-9f04-00667bacb002-tls-certs\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.287722 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.287515 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-home\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.287722 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.287550 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm7q5\" (UniqueName: \"kubernetes.io/projected/0444d15e-1c92-4f39-9f04-00667bacb002-kube-api-access-jm7q5\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.287722 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.287625 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-dshm\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.287722 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.287700 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-tmp-dir\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.287981 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.287739 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-model-cache\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.388399 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.388366 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-home\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.388596 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.388412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm7q5\" (UniqueName: \"kubernetes.io/projected/0444d15e-1c92-4f39-9f04-00667bacb002-kube-api-access-jm7q5\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.388596 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.388446 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-dshm\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.388596 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.388502 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-tmp-dir\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.388596 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.388536 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-model-cache\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.388875 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.388609 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0444d15e-1c92-4f39-9f04-00667bacb002-tls-certs\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.388875 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.388841 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-home\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.388992 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.388937 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-model-cache\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.389083 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.389065 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-tmp-dir\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.391172 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.391147 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-dshm\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.391517 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.391495 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0444d15e-1c92-4f39-9f04-00667bacb002-tls-certs\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.398055 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.398030 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm7q5\" (UniqueName: \"kubernetes.io/projected/0444d15e-1c92-4f39-9f04-00667bacb002-kube-api-access-jm7q5\") pod \"rate-test-model-f556c1-kserve-67d44b4b99-5kdg6\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.571762 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.571723 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:31.713868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:31.713823 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6"] Jul 24 18:24:31.717025 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:24:31.716999 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0444d15e_1c92_4f39_9f04_00667bacb002.slice/crio-61ad67f3dc550b2a2f552b95e1d3d47273a4cb39c3e658e1d0d491766afb30a0 WatchSource:0}: Error finding container 61ad67f3dc550b2a2f552b95e1d3d47273a4cb39c3e658e1d0d491766afb30a0: Status 404 returned error can't find the container with id 61ad67f3dc550b2a2f552b95e1d3d47273a4cb39c3e658e1d0d491766afb30a0 Jul 24 18:24:32.157670 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:32.157632 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" event={"ID":"0444d15e-1c92-4f39-9f04-00667bacb002","Type":"ContainerStarted","Data":"96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65"} Jul 24 18:24:32.157670 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:32.157673 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" event={"ID":"0444d15e-1c92-4f39-9f04-00667bacb002","Type":"ContainerStarted","Data":"61ad67f3dc550b2a2f552b95e1d3d47273a4cb39c3e658e1d0d491766afb30a0"} Jul 24 18:24:32.197460 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:32.197407 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" podStartSLOduration=1.197389318 podStartE2EDuration="1.197389318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:24:32.194910796 +0000 UTC m=+2497.234704849" watchObservedRunningTime="2026-07-24 18:24:32.197389318 +0000 UTC m=+2497.237183358" Jul 24 18:24:34.056670 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:34.056632 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:24:41.572954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:41.572916 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:41.573351 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:41.572971 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:41.585604 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:41.585578 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:42.217353 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:42.217317 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:24:57.666841 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:24:57.666799 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:25:10.157557 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:10.157513 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:25:15.332890 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:15.332857 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch"] Jul 24 18:25:16.120595 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:16.120561 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6"] Jul 24 18:25:18.153757 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:18.153721 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:25:23.276904 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.276854 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:23.280315 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.280299 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:23.283279 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.283254 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-ksjv4\"" Jul 24 18:25:23.284759 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.284692 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:23.345969 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.345925 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2zsc\" (UniqueName: \"kubernetes.io/projected/043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3-kube-api-access-z2zsc\") pod \"test-curl-7539-139713846750352\" (UID: \"043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3\") " pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:23.446821 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.446769 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2zsc\" (UniqueName: \"kubernetes.io/projected/043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3-kube-api-access-z2zsc\") pod \"test-curl-7539-139713846750352\" (UID: \"043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3\") " pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:23.455589 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.455557 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2zsc\" (UniqueName: \"kubernetes.io/projected/043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3-kube-api-access-z2zsc\") pod \"test-curl-7539-139713846750352\" (UID: \"043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3\") " pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:23.590702 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.590671 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:23.727739 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.727715 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:23.729670 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:25:23.729642 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod043c993e_fb21_4c58_9bd6_5bcdfcfbfcd3.slice/crio-f3715b43ca7d658345b015c3fc760cb9a34ada70e6bd2d45dac5f495d0834683 WatchSource:0}: Error finding container f3715b43ca7d658345b015c3fc760cb9a34ada70e6bd2d45dac5f495d0834683: Status 404 returned error can't find the container with id f3715b43ca7d658345b015c3fc760cb9a34ada70e6bd2d45dac5f495d0834683 Jul 24 18:25:23.731097 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:23.731077 2559 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 24 18:25:24.406572 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:24.406537 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7539-139713846750352" event={"ID":"043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3","Type":"ContainerStarted","Data":"f3715b43ca7d658345b015c3fc760cb9a34ada70e6bd2d45dac5f495d0834683"} Jul 24 18:25:25.412933 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:25.412899 2559 generic.go:356] "Generic (PLEG): container finished" podID="043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3" containerID="61ef7939302ca8c937af0d9df40f4ef865b7a970025cc633f9d8e4db54c7fbda" exitCode=0 Jul 24 18:25:25.413323 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:25.412969 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7539-139713846750352" event={"ID":"043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3","Type":"ContainerDied","Data":"61ef7939302ca8c937af0d9df40f4ef865b7a970025cc633f9d8e4db54c7fbda"} Jul 24 18:25:26.553502 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.553479 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:26.573725 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.573694 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z2zsc\" (UniqueName: \"kubernetes.io/projected/043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3-kube-api-access-z2zsc\") pod \"043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3\" (UID: \"043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3\") " Jul 24 18:25:26.576128 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.576099 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3-kube-api-access-z2zsc" (OuterVolumeSpecName: "kube-api-access-z2zsc") pod "043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3" (UID: "043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3"). InnerVolumeSpecName "kube-api-access-z2zsc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:25:26.590662 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.590640 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7539-139713846750352_043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3/test-curl-7539-139713846750352/0.log" Jul 24 18:25:26.614872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.614846 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:26.619145 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.619124 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:26.675856 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.675746 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2zsc\" (UniqueName: \"kubernetes.io/projected/043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3-kube-api-access-z2zsc\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:26.745688 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.745658 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:26.746020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.746008 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3" containerName="test-curl-7539-139713846750352" Jul 24 18:25:26.746114 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.746022 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3" containerName="test-curl-7539-139713846750352" Jul 24 18:25:26.746114 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.746089 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3" containerName="test-curl-7539-139713846750352" Jul 24 18:25:26.748929 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.748914 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:26.757816 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.757770 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:26.776684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.776657 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vl6w\" (UniqueName: \"kubernetes.io/projected/ca58cd49-7ce5-459d-afc7-d6dfebf05df8-kube-api-access-4vl6w\") pod \"test-curl-7539-139713846750352\" (UID: \"ca58cd49-7ce5-459d-afc7-d6dfebf05df8\") " pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:26.878002 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.877969 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vl6w\" (UniqueName: \"kubernetes.io/projected/ca58cd49-7ce5-459d-afc7-d6dfebf05df8-kube-api-access-4vl6w\") pod \"test-curl-7539-139713846750352\" (UID: \"ca58cd49-7ce5-459d-afc7-d6dfebf05df8\") " pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:26.887418 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:26.887387 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vl6w\" (UniqueName: \"kubernetes.io/projected/ca58cd49-7ce5-459d-afc7-d6dfebf05df8-kube-api-access-4vl6w\") pod \"test-curl-7539-139713846750352\" (UID: \"ca58cd49-7ce5-459d-afc7-d6dfebf05df8\") " pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:27.058304 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:27.058211 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:27.193412 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:27.193384 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:27.195664 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:25:27.195643 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podca58cd49_7ce5_459d_afc7_d6dfebf05df8.slice/crio-7a32adb4917f9783c3d5b1beef36049f18ed851615c874bfe6770e5608787f35 WatchSource:0}: Error finding container 7a32adb4917f9783c3d5b1beef36049f18ed851615c874bfe6770e5608787f35: Status 404 returned error can't find the container with id 7a32adb4917f9783c3d5b1beef36049f18ed851615c874bfe6770e5608787f35 Jul 24 18:25:27.423528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:27.423499 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:27.423731 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:27.423551 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3715b43ca7d658345b015c3fc760cb9a34ada70e6bd2d45dac5f495d0834683" Jul 24 18:25:27.424661 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:27.424627 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7539-139713846750352" event={"ID":"ca58cd49-7ce5-459d-afc7-d6dfebf05df8","Type":"ContainerStarted","Data":"7a32adb4917f9783c3d5b1beef36049f18ed851615c874bfe6770e5608787f35"} Jul 24 18:25:27.435399 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:27.435369 2559 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7539-139713846750352" oldPodUID="043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" Jul 24 18:25:27.494059 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:27.494033 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3" path="/var/lib/kubelet/pods/043c993e-fb21-4c58-9bd6-5bcdfcfbfcd3/volumes" Jul 24 18:25:28.430119 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:28.430081 2559 generic.go:356] "Generic (PLEG): container finished" podID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" containerID="767e54d4ddb2742fa1a91a5d3c37b5b409a858b0355252d0abea2472fa6e94f1" exitCode=0 Jul 24 18:25:28.430510 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:28.430161 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7539-139713846750352" event={"ID":"ca58cd49-7ce5-459d-afc7-d6dfebf05df8","Type":"ContainerDied","Data":"767e54d4ddb2742fa1a91a5d3c37b5b409a858b0355252d0abea2472fa6e94f1"} Jul 24 18:25:29.575463 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.575441 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:29.597282 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.597254 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7539-139713846750352_ca58cd49-7ce5-459d-afc7-d6dfebf05df8/test-curl-7539-139713846750352/0.log" Jul 24 18:25:29.601472 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.601452 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vl6w\" (UniqueName: \"kubernetes.io/projected/ca58cd49-7ce5-459d-afc7-d6dfebf05df8-kube-api-access-4vl6w\") pod \"ca58cd49-7ce5-459d-afc7-d6dfebf05df8\" (UID: \"ca58cd49-7ce5-459d-afc7-d6dfebf05df8\") " Jul 24 18:25:29.603685 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.603656 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca58cd49-7ce5-459d-afc7-d6dfebf05df8-kube-api-access-4vl6w" (OuterVolumeSpecName: "kube-api-access-4vl6w") pod "ca58cd49-7ce5-459d-afc7-d6dfebf05df8" (UID: "ca58cd49-7ce5-459d-afc7-d6dfebf05df8"). InnerVolumeSpecName "kube-api-access-4vl6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:25:29.623899 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.623872 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:29.631843 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.631821 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7539-139713846750352"] Jul 24 18:25:29.702803 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.702696 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vl6w\" (UniqueName: \"kubernetes.io/projected/ca58cd49-7ce5-459d-afc7-d6dfebf05df8-kube-api-access-4vl6w\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:29.755975 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.755947 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:29.756279 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.756266 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" containerName="test-curl-7539-139713846750352" Jul 24 18:25:29.756317 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.756281 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" containerName="test-curl-7539-139713846750352" Jul 24 18:25:29.756366 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.756344 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" containerName="test-curl-7539-139713846750352" Jul 24 18:25:29.759161 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.759145 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:29.776721 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.776690 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:29.803727 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.803689 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vw5sj\" (UniqueName: \"kubernetes.io/projected/8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc-kube-api-access-vw5sj\") pod \"test-curl-7539-139713849188400\" (UID: \"8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc\") " pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:29.904540 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.904498 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vw5sj\" (UniqueName: \"kubernetes.io/projected/8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc-kube-api-access-vw5sj\") pod \"test-curl-7539-139713849188400\" (UID: \"8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc\") " pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:29.913091 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:29.913058 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vw5sj\" (UniqueName: \"kubernetes.io/projected/8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc-kube-api-access-vw5sj\") pod \"test-curl-7539-139713849188400\" (UID: \"8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc\") " pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:30.071290 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.071262 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:30.206196 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.206165 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:30.208583 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:25:30.208558 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bfef5fe_bb35_4dc7_844b_ebfb7c92b2dc.slice/crio-d628735def323b3930c3193225f1e93b7d547ee22472bacf18b2a1c2d7f1df50 WatchSource:0}: Error finding container d628735def323b3930c3193225f1e93b7d547ee22472bacf18b2a1c2d7f1df50: Status 404 returned error can't find the container with id d628735def323b3930c3193225f1e93b7d547ee22472bacf18b2a1c2d7f1df50 Jul 24 18:25:30.352765 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.352658 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" podUID="e42143eb-1659-46cb-9c51-47de4a2a96f7" containerName="main" containerID="cri-o://54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a" gracePeriod=45 Jul 24 18:25:30.441030 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.441000 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:30.441202 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.441001 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a32adb4917f9783c3d5b1beef36049f18ed851615c874bfe6770e5608787f35" Jul 24 18:25:30.442202 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.442176 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7539-139713849188400" event={"ID":"8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc","Type":"ContainerStarted","Data":"d628735def323b3930c3193225f1e93b7d547ee22472bacf18b2a1c2d7f1df50"} Jul 24 18:25:30.451123 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:25:30.451097 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7539-139713846750352\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:30.654346 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.654318 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:25:30.673184 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:25:30.673135 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7539-139713846750352\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:30.709545 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.709503 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2sw2n\" (UniqueName: \"kubernetes.io/projected/e42143eb-1659-46cb-9c51-47de4a2a96f7-kube-api-access-2sw2n\") pod \"e42143eb-1659-46cb-9c51-47de4a2a96f7\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " Jul 24 18:25:30.709724 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.709561 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-home\") pod \"e42143eb-1659-46cb-9c51-47de4a2a96f7\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " Jul 24 18:25:30.709724 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.709600 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42143eb-1659-46cb-9c51-47de4a2a96f7-tls-certs\") pod \"e42143eb-1659-46cb-9c51-47de4a2a96f7\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " Jul 24 18:25:30.709724 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.709618 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-dshm\") pod \"e42143eb-1659-46cb-9c51-47de4a2a96f7\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " Jul 24 18:25:30.709724 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.709644 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-tmp-dir\") pod \"e42143eb-1659-46cb-9c51-47de4a2a96f7\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " Jul 24 18:25:30.709724 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.709676 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-model-cache\") pod \"e42143eb-1659-46cb-9c51-47de4a2a96f7\" (UID: \"e42143eb-1659-46cb-9c51-47de4a2a96f7\") " Jul 24 18:25:30.710039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.709905 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-home" (OuterVolumeSpecName: "home") pod "e42143eb-1659-46cb-9c51-47de4a2a96f7" (UID: "e42143eb-1659-46cb-9c51-47de4a2a96f7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:25:30.710039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.709963 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e42143eb-1659-46cb-9c51-47de4a2a96f7" (UID: "e42143eb-1659-46cb-9c51-47de4a2a96f7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:25:30.710039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.710019 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-model-cache" (OuterVolumeSpecName: "model-cache") pod "e42143eb-1659-46cb-9c51-47de4a2a96f7" (UID: "e42143eb-1659-46cb-9c51-47de4a2a96f7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:25:30.711870 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.711844 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-dshm" (OuterVolumeSpecName: "dshm") pod "e42143eb-1659-46cb-9c51-47de4a2a96f7" (UID: "e42143eb-1659-46cb-9c51-47de4a2a96f7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:25:30.711870 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.711850 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e42143eb-1659-46cb-9c51-47de4a2a96f7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e42143eb-1659-46cb-9c51-47de4a2a96f7" (UID: "e42143eb-1659-46cb-9c51-47de4a2a96f7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:25:30.712005 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.711937 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e42143eb-1659-46cb-9c51-47de4a2a96f7-kube-api-access-2sw2n" (OuterVolumeSpecName: "kube-api-access-2sw2n") pod "e42143eb-1659-46cb-9c51-47de4a2a96f7" (UID: "e42143eb-1659-46cb-9c51-47de4a2a96f7"). InnerVolumeSpecName "kube-api-access-2sw2n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:25:30.810853 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.810821 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sw2n\" (UniqueName: \"kubernetes.io/projected/e42143eb-1659-46cb-9c51-47de4a2a96f7-kube-api-access-2sw2n\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:30.810853 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.810848 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:30.810853 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.810857 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e42143eb-1659-46cb-9c51-47de4a2a96f7-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:30.811093 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.810867 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:30.811093 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.810875 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:30.811093 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:30.810882 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e42143eb-1659-46cb-9c51-47de4a2a96f7-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:31.143840 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.143749 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" podUID="0444d15e-1c92-4f39-9f04-00667bacb002" containerName="main" containerID="cri-o://96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65" gracePeriod=45 Jul 24 18:25:31.397386 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.397312 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:25:31.416675 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.416633 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-dshm\") pod \"0444d15e-1c92-4f39-9f04-00667bacb002\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " Jul 24 18:25:31.416872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.416699 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-home\") pod \"0444d15e-1c92-4f39-9f04-00667bacb002\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " Jul 24 18:25:31.416872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.416732 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jm7q5\" (UniqueName: \"kubernetes.io/projected/0444d15e-1c92-4f39-9f04-00667bacb002-kube-api-access-jm7q5\") pod \"0444d15e-1c92-4f39-9f04-00667bacb002\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " Jul 24 18:25:31.416872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.416803 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-model-cache\") pod \"0444d15e-1c92-4f39-9f04-00667bacb002\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " Jul 24 18:25:31.416872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.416848 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0444d15e-1c92-4f39-9f04-00667bacb002-tls-certs\") pod \"0444d15e-1c92-4f39-9f04-00667bacb002\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " Jul 24 18:25:31.417097 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.416905 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-tmp-dir\") pod \"0444d15e-1c92-4f39-9f04-00667bacb002\" (UID: \"0444d15e-1c92-4f39-9f04-00667bacb002\") " Jul 24 18:25:31.418223 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.417267 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-model-cache" (OuterVolumeSpecName: "model-cache") pod "0444d15e-1c92-4f39-9f04-00667bacb002" (UID: "0444d15e-1c92-4f39-9f04-00667bacb002"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:25:31.418223 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.417407 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-home" (OuterVolumeSpecName: "home") pod "0444d15e-1c92-4f39-9f04-00667bacb002" (UID: "0444d15e-1c92-4f39-9f04-00667bacb002"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:25:31.418223 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:25:31.417449 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7539-139713846750352\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:31.418223 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.417635 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0444d15e-1c92-4f39-9f04-00667bacb002" (UID: "0444d15e-1c92-4f39-9f04-00667bacb002"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:25:31.418223 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.417822 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:31.418223 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.417837 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:31.418223 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.417846 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:31.419482 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.419437 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-dshm" (OuterVolumeSpecName: "dshm") pod "0444d15e-1c92-4f39-9f04-00667bacb002" (UID: "0444d15e-1c92-4f39-9f04-00667bacb002"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:25:31.419655 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.419613 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0444d15e-1c92-4f39-9f04-00667bacb002-kube-api-access-jm7q5" (OuterVolumeSpecName: "kube-api-access-jm7q5") pod "0444d15e-1c92-4f39-9f04-00667bacb002" (UID: "0444d15e-1c92-4f39-9f04-00667bacb002"). InnerVolumeSpecName "kube-api-access-jm7q5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:25:31.420247 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.420215 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0444d15e-1c92-4f39-9f04-00667bacb002-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0444d15e-1c92-4f39-9f04-00667bacb002" (UID: "0444d15e-1c92-4f39-9f04-00667bacb002"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:25:31.450050 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.450018 2559 generic.go:356] "Generic (PLEG): container finished" podID="8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc" containerID="77b6d6e3fac8b785234308b26785580dc1cb8c3b4d992d7d133d80b026d644eb" exitCode=0 Jul 24 18:25:31.450234 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.450096 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7539-139713849188400" event={"ID":"8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc","Type":"ContainerDied","Data":"77b6d6e3fac8b785234308b26785580dc1cb8c3b4d992d7d133d80b026d644eb"} Jul 24 18:25:31.451369 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.451349 2559 generic.go:356] "Generic (PLEG): container finished" podID="0444d15e-1c92-4f39-9f04-00667bacb002" containerID="96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65" exitCode=0 Jul 24 18:25:31.451477 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.451387 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" Jul 24 18:25:31.451477 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.451421 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" event={"ID":"0444d15e-1c92-4f39-9f04-00667bacb002","Type":"ContainerDied","Data":"96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65"} Jul 24 18:25:31.451477 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.451459 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6" event={"ID":"0444d15e-1c92-4f39-9f04-00667bacb002","Type":"ContainerDied","Data":"61ad67f3dc550b2a2f552b95e1d3d47273a4cb39c3e658e1d0d491766afb30a0"} Jul 24 18:25:31.451640 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.451479 2559 scope.go:117] "RemoveContainer" containerID="96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65" Jul 24 18:25:31.452435 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:25:31.452410 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7539-139713846750352\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:31.452960 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.452942 2559 generic.go:356] "Generic (PLEG): container finished" podID="e42143eb-1659-46cb-9c51-47de4a2a96f7" containerID="54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a" exitCode=0 Jul 24 18:25:31.453051 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.452981 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" Jul 24 18:25:31.453051 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.452996 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" event={"ID":"e42143eb-1659-46cb-9c51-47de4a2a96f7","Type":"ContainerDied","Data":"54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a"} Jul 24 18:25:31.453051 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.453017 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch" event={"ID":"e42143eb-1659-46cb-9c51-47de4a2a96f7","Type":"ContainerDied","Data":"0faea21ef5ae146c7c254a008a3e4b2cc18db7a71bed3d9d049a602e127586b6"} Jul 24 18:25:31.463325 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.463302 2559 scope.go:117] "RemoveContainer" containerID="96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65" Jul 24 18:25:31.463642 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:25:31.463619 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65\": container with ID starting with 96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65 not found: ID does not exist" containerID="96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65" Jul 24 18:25:31.463722 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.463650 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65"} err="failed to get container status \"96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65\": rpc error: code = NotFound desc = could not find container \"96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65\": container with ID starting with 96ac61e8c4d84f7911cde35a3441c38c022cdbcf8eabde4747b2e747beceaf65 not found: ID does not exist" Jul 24 18:25:31.463722 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.463666 2559 scope.go:117] "RemoveContainer" containerID="54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a" Jul 24 18:25:31.472263 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.472102 2559 scope.go:117] "RemoveContainer" containerID="54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a" Jul 24 18:25:31.472397 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:25:31.472379 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a\": container with ID starting with 54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a not found: ID does not exist" containerID="54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a" Jul 24 18:25:31.472444 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.472405 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a"} err="failed to get container status \"54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a\": rpc error: code = NotFound desc = could not find container \"54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a\": container with ID starting with 54a6c4e5545c062af60f80b47f2cfc0f22cce68e3d39b0e98876514f374d315a not found: ID does not exist" Jul 24 18:25:31.482179 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.482157 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6"] Jul 24 18:25:31.486314 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.486292 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f556c1/rate-test-model-f556c1-kserve-67d44b4b99-5kdg6"] Jul 24 18:25:31.487082 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:25:31.487063 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7539-139713846750352\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" pod="opendatahub/test-curl-7539-139713846750352" Jul 24 18:25:31.494588 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.494565 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0444d15e-1c92-4f39-9f04-00667bacb002" path="/var/lib/kubelet/pods/0444d15e-1c92-4f39-9f04-00667bacb002/volumes" Jul 24 18:25:31.494906 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.494894 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca58cd49-7ce5-459d-afc7-d6dfebf05df8" path="/var/lib/kubelet/pods/ca58cd49-7ce5-459d-afc7-d6dfebf05df8/volumes" Jul 24 18:25:31.500207 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.500181 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch"] Jul 24 18:25:31.504331 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.504311 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-2542df/rate-test-model-2542df-kserve-db7667679-cvwch"] Jul 24 18:25:31.519187 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.519166 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0444d15e-1c92-4f39-9f04-00667bacb002-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:31.519263 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.519190 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jm7q5\" (UniqueName: \"kubernetes.io/projected/0444d15e-1c92-4f39-9f04-00667bacb002-kube-api-access-jm7q5\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:31.519263 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:31.519205 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0444d15e-1c92-4f39-9f04-00667bacb002-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:32.595558 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.595534 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:32.617127 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.617098 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7539-139713849188400_8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc/test-curl-7539-139713849188400/0.log" Jul 24 18:25:32.628272 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.628247 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vw5sj\" (UniqueName: \"kubernetes.io/projected/8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc-kube-api-access-vw5sj\") pod \"8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc\" (UID: \"8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc\") " Jul 24 18:25:32.630602 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.630570 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc-kube-api-access-vw5sj" (OuterVolumeSpecName: "kube-api-access-vw5sj") pod "8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc" (UID: "8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc"). InnerVolumeSpecName "kube-api-access-vw5sj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:25:32.640855 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.640826 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:32.644630 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.644603 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:32.729573 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.729480 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vw5sj\" (UniqueName: \"kubernetes.io/projected/8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc-kube-api-access-vw5sj\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:32.765987 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.765953 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:32.766324 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766311 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0444d15e-1c92-4f39-9f04-00667bacb002" containerName="main" Jul 24 18:25:32.766378 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766327 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="0444d15e-1c92-4f39-9f04-00667bacb002" containerName="main" Jul 24 18:25:32.766378 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766335 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc" containerName="test-curl-7539-139713849188400" Jul 24 18:25:32.766378 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766341 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc" containerName="test-curl-7539-139713849188400" Jul 24 18:25:32.766378 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766358 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e42143eb-1659-46cb-9c51-47de4a2a96f7" containerName="main" Jul 24 18:25:32.766378 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766363 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="e42143eb-1659-46cb-9c51-47de4a2a96f7" containerName="main" Jul 24 18:25:32.766551 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766417 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="e42143eb-1659-46cb-9c51-47de4a2a96f7" containerName="main" Jul 24 18:25:32.766551 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766424 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc" containerName="test-curl-7539-139713849188400" Jul 24 18:25:32.766551 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.766431 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="0444d15e-1c92-4f39-9f04-00667bacb002" containerName="main" Jul 24 18:25:32.769338 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.769322 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:32.776993 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.776970 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:32.830449 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.830419 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4pkd\" (UniqueName: \"kubernetes.io/projected/8ba98d40-5ea5-4581-99ab-bf16fc5b7145-kube-api-access-t4pkd\") pod \"test-curl-7539-139713849188400\" (UID: \"8ba98d40-5ea5-4581-99ab-bf16fc5b7145\") " pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:32.931075 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.931036 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4pkd\" (UniqueName: \"kubernetes.io/projected/8ba98d40-5ea5-4581-99ab-bf16fc5b7145-kube-api-access-t4pkd\") pod \"test-curl-7539-139713849188400\" (UID: \"8ba98d40-5ea5-4581-99ab-bf16fc5b7145\") " pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:32.939066 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:32.939036 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4pkd\" (UniqueName: \"kubernetes.io/projected/8ba98d40-5ea5-4581-99ab-bf16fc5b7145-kube-api-access-t4pkd\") pod \"test-curl-7539-139713849188400\" (UID: \"8ba98d40-5ea5-4581-99ab-bf16fc5b7145\") " pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:33.079090 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:33.079063 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:33.211447 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:33.211419 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:33.213476 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:25:33.213452 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ba98d40_5ea5_4581_99ab_bf16fc5b7145.slice/crio-c64d160c28283c18ba953d403787a50b7d23c792bed5e7a564fb2571321ef610 WatchSource:0}: Error finding container c64d160c28283c18ba953d403787a50b7d23c792bed5e7a564fb2571321ef610: Status 404 returned error can't find the container with id c64d160c28283c18ba953d403787a50b7d23c792bed5e7a564fb2571321ef610 Jul 24 18:25:33.466138 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:33.466058 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d628735def323b3930c3193225f1e93b7d547ee22472bacf18b2a1c2d7f1df50" Jul 24 18:25:33.466138 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:33.466055 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:33.467221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:33.467193 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7539-139713849188400" event={"ID":"8ba98d40-5ea5-4581-99ab-bf16fc5b7145","Type":"ContainerStarted","Data":"c64d160c28283c18ba953d403787a50b7d23c792bed5e7a564fb2571321ef610"} Jul 24 18:25:33.481899 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:33.481870 2559 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7539-139713849188400" oldPodUID="8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc" podUID="8ba98d40-5ea5-4581-99ab-bf16fc5b7145" Jul 24 18:25:33.495861 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:33.495831 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc" path="/var/lib/kubelet/pods/8bfef5fe-bb35-4dc7-844b-ebfb7c92b2dc/volumes" Jul 24 18:25:33.496298 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:33.496279 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e42143eb-1659-46cb-9c51-47de4a2a96f7" path="/var/lib/kubelet/pods/e42143eb-1659-46cb-9c51-47de4a2a96f7/volumes" Jul 24 18:25:34.473546 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:34.473515 2559 generic.go:356] "Generic (PLEG): container finished" podID="8ba98d40-5ea5-4581-99ab-bf16fc5b7145" containerID="3dcdd27cd0dbad4551b89a289eb9b752e7ab7ae80299f346045aa2215ec39fe1" exitCode=0 Jul 24 18:25:34.473970 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:34.473608 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7539-139713849188400" event={"ID":"8ba98d40-5ea5-4581-99ab-bf16fc5b7145","Type":"ContainerDied","Data":"3dcdd27cd0dbad4551b89a289eb9b752e7ab7ae80299f346045aa2215ec39fe1"} Jul 24 18:25:35.621828 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:35.621801 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:35.642899 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:35.642878 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7539-139713849188400_8ba98d40-5ea5-4581-99ab-bf16fc5b7145/test-curl-7539-139713849188400/0.log" Jul 24 18:25:35.658149 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:35.658125 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4pkd\" (UniqueName: \"kubernetes.io/projected/8ba98d40-5ea5-4581-99ab-bf16fc5b7145-kube-api-access-t4pkd\") pod \"8ba98d40-5ea5-4581-99ab-bf16fc5b7145\" (UID: \"8ba98d40-5ea5-4581-99ab-bf16fc5b7145\") " Jul 24 18:25:35.660313 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:35.660284 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba98d40-5ea5-4581-99ab-bf16fc5b7145-kube-api-access-t4pkd" (OuterVolumeSpecName: "kube-api-access-t4pkd") pod "8ba98d40-5ea5-4581-99ab-bf16fc5b7145" (UID: "8ba98d40-5ea5-4581-99ab-bf16fc5b7145"). InnerVolumeSpecName "kube-api-access-t4pkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:25:35.669582 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:35.669552 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:35.673318 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:35.673298 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7539-139713849188400"] Jul 24 18:25:35.759045 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:35.758963 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4pkd\" (UniqueName: \"kubernetes.io/projected/8ba98d40-5ea5-4581-99ab-bf16fc5b7145-kube-api-access-t4pkd\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:25:36.464823 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.464773 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh"] Jul 24 18:25:36.471549 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.471516 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba98d40-5ea5-4581-99ab-bf16fc5b7145" containerName="test-curl-7539-139713849188400" Jul 24 18:25:36.471733 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.471545 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba98d40-5ea5-4581-99ab-bf16fc5b7145" containerName="test-curl-7539-139713849188400" Jul 24 18:25:36.471733 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.471706 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba98d40-5ea5-4581-99ab-bf16fc5b7145" containerName="test-curl-7539-139713849188400" Jul 24 18:25:36.474652 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.474633 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.477744 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.477726 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-d91e94-openshift-default-dockercfg-jh7lw\"" Jul 24 18:25:36.482123 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.482098 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh"] Jul 24 18:25:36.487473 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.487452 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64d160c28283c18ba953d403787a50b7d23c792bed5e7a564fb2571321ef610" Jul 24 18:25:36.487473 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.487468 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:36.498011 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:25:36.497984 2559 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7539-139713849188400\" is forbidden: User \"system:node:ip-10-0-141-191.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-191.ec2.internal' and this object" podUID="8ba98d40-5ea5-4581-99ab-bf16fc5b7145" pod="opendatahub/test-curl-7539-139713849188400" Jul 24 18:25:36.565912 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.565884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-credential-socket\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.566095 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.565938 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-envoy\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.566095 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.565963 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e18dfc-e25b-47b7-b493-a917cadd743c-istiod-ca-cert\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.566095 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.566058 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-certs\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.566223 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.566098 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-token\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.566270 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.566215 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24dkt\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-kube-api-access-24dkt\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.566310 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.566282 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-socket\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.566354 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.566314 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-data\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.566354 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.566349 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-podinfo\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.667689 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.667648 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-certs\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.667689 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.667693 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-token\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.667879 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24dkt\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-kube-api-access-24dkt\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.667947 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-socket\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.667980 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-data\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.668007 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-podinfo\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.668043 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-credential-socket\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.668058 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-certs\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.668143 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-envoy\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668204 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.668188 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e18dfc-e25b-47b7-b493-a917cadd743c-istiod-ca-cert\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668643 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.668316 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-socket\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668643 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.668340 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-credential-socket\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.668643 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.668443 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-data\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.669054 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.669031 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e18dfc-e25b-47b7-b493-a917cadd743c-istiod-ca-cert\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.670519 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.670492 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-envoy\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.670720 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.670704 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-podinfo\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.678096 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.678078 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-token\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.678327 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.678305 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24dkt\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-kube-api-access-24dkt\") pod \"e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.787103 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.787001 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:36.930307 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.930283 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh"] Jul 24 18:25:36.932439 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:25:36.932406 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79e18dfc_e25b_47b7_b493_a917cadd743c.slice/crio-c05837af70b670d85955fd3a12cd74b097e6a070c95a59019e5fd06d732ef08a WatchSource:0}: Error finding container c05837af70b670d85955fd3a12cd74b097e6a070c95a59019e5fd06d732ef08a: Status 404 returned error can't find the container with id c05837af70b670d85955fd3a12cd74b097e6a070c95a59019e5fd06d732ef08a Jul 24 18:25:36.934880 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.934836 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:36.934962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.934938 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:36.934997 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:36.934975 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:37.494326 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:37.494286 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba98d40-5ea5-4581-99ab-bf16fc5b7145" path="/var/lib/kubelet/pods/8ba98d40-5ea5-4581-99ab-bf16fc5b7145/volumes" Jul 24 18:25:37.494941 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:37.494912 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" event={"ID":"79e18dfc-e25b-47b7-b493-a917cadd743c","Type":"ContainerStarted","Data":"352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6"} Jul 24 18:25:37.495082 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:37.494962 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" event={"ID":"79e18dfc-e25b-47b7-b493-a917cadd743c","Type":"ContainerStarted","Data":"c05837af70b670d85955fd3a12cd74b097e6a070c95a59019e5fd06d732ef08a"} Jul 24 18:25:37.518331 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:37.518281 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" podStartSLOduration=1.518265654 podStartE2EDuration="1.518265654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:25:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:25:37.515702271 +0000 UTC m=+2562.555496312" watchObservedRunningTime="2026-07-24 18:25:37.518265654 +0000 UTC m=+2562.558059695" Jul 24 18:25:37.787367 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:37.787290 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:37.792189 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:37.792163 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:38.500225 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:38.500188 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:38.501224 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:38.501204 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:43.356047 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:43.355996 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:25:53.502307 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:53.502277 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 18:25:53.502743 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:53.502329 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 18:25:53.574050 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:53.574020 2559 generic.go:356] "Generic (PLEG): container finished" podID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerID="352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6" exitCode=0 Jul 24 18:25:53.574223 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:53.574095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" event={"ID":"79e18dfc-e25b-47b7-b493-a917cadd743c","Type":"ContainerDied","Data":"352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6"} Jul 24 18:25:53.574494 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:53.574481 2559 scope.go:117] "RemoveContainer" containerID="352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6" Jul 24 18:25:53.575528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:53.575502 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:53.575622 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:53.575582 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:53.575685 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:53.575629 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:54.583764 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:54.583728 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" event={"ID":"79e18dfc-e25b-47b7-b493-a917cadd743c","Type":"ContainerStarted","Data":"c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675"} Jul 24 18:25:54.787804 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:54.787739 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:54.788125 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:54.788097 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 18:25:54.788210 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:54.788170 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 18:25:55.275530 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.275496 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts"] Jul 24 18:25:55.280373 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.280354 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.295507 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.290220 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-651f67-openshift-default-dockercfg-2jklf\"" Jul 24 18:25:55.315032 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.315000 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts"] Jul 24 18:25:55.450504 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450454 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-949n8\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-kube-api-access-949n8\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.450711 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450514 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-podinfo\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.450711 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450563 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-credential-socket\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.450711 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450611 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-data\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.450711 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450678 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istiod-ca-cert\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.450711 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450708 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-envoy\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.450999 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450732 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-token\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.450999 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-certs\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.450999 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.450837 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-socket\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552172 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552080 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-data\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552172 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552144 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istiod-ca-cert\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552172 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552165 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-envoy\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552461 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552181 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-token\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552461 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552203 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-certs\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552461 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552320 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-socket\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552461 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552380 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-949n8\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-kube-api-access-949n8\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552461 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552412 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-podinfo\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552461 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552439 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-credential-socket\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552744 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552572 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-data\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552744 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552593 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-certs\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.552968 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552944 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-credential-socket\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.553037 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552968 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istiod-ca-cert\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.553037 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.552989 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-socket\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.554673 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.554655 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-envoy\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.555112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.555092 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-podinfo\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.560825 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.560775 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-token\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.561119 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.561098 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-949n8\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-kube-api-access-949n8\") pod \"e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.618947 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.618916 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-651f67-openshift-default-dockercfg-2jklf\"" Jul 24 18:25:55.627154 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.627113 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:55.781112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.781085 2559 scope.go:117] "RemoveContainer" containerID="b1dafd7d797e21d3411d154b7a94c2955f28a5d7354031c38b9271545abb1037" Jul 24 18:25:55.812079 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.812048 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts"] Jul 24 18:25:55.812984 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.812240 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 18:25:55.812984 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.812295 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 18:25:55.815251 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:25:55.815213 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81fcdaba_b5bb_4265_9949_fa8a8d0a9191.slice/crio-90d5a2bcbd1dbb9440319bb058b0848a3421abb161363230cea447f7f0895d0d WatchSource:0}: Error finding container 90d5a2bcbd1dbb9440319bb058b0848a3421abb161363230cea447f7f0895d0d: Status 404 returned error can't find the container with id 90d5a2bcbd1dbb9440319bb058b0848a3421abb161363230cea447f7f0895d0d Jul 24 18:25:55.818037 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.818013 2559 scope.go:117] "RemoveContainer" containerID="6333ac0959ebb5f24f12ccfe6f7e124accea58b4c6ea506ce4adc3498cea8238" Jul 24 18:25:55.818103 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.818032 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:55.818103 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.818097 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:55.818191 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:55.818130 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:25:56.610855 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:56.610808 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" event={"ID":"81fcdaba-b5bb-4265-9949-fa8a8d0a9191","Type":"ContainerStarted","Data":"7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31"} Jul 24 18:25:56.611072 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:56.610901 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" event={"ID":"81fcdaba-b5bb-4265-9949-fa8a8d0a9191","Type":"ContainerStarted","Data":"90d5a2bcbd1dbb9440319bb058b0848a3421abb161363230cea447f7f0895d0d"} Jul 24 18:25:56.627861 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:56.627829 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:56.632805 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:56.632762 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:56.637666 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:56.637618 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" podStartSLOduration=1.6375934330000002 podStartE2EDuration="1.637593433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:25:56.636723174 +0000 UTC m=+2581.676517241" watchObservedRunningTime="2026-07-24 18:25:56.637593433 +0000 UTC m=+2581.677387546" Jul 24 18:25:56.788051 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:56.788004 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 18:25:56.788264 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:56.788075 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 18:25:57.616025 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:57.615994 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:57.617182 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:57.617161 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:25:57.788153 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:57.788113 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 18:25:57.788660 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:57.788182 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 18:25:58.788361 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:58.788315 2559 patch_prober.go:28] interesting pod/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" start-of-body= Jul 24 18:25:58.788893 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:58.788382 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.56:15021/healthz/ready\": dial tcp 10.132.0.56:15021: connect: connection refused" Jul 24 18:25:59.792144 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:59.792114 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:59.792516 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:59.792425 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:25:59.793188 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:25:59.793172 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:26:03.451840 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:03.451808 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:26:13.280274 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:13.279364 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:57244->10.132.0.57:15021: read: connection reset by peer" start-of-body= Jul 24 18:26:13.280274 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:13.279435 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": read tcp 10.132.0.2:57244->10.132.0.57:15021: read: connection reset by peer" Jul 24 18:26:13.706745 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:13.706713 2559 generic.go:356] "Generic (PLEG): container finished" podID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerID="7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31" exitCode=0 Jul 24 18:26:13.706934 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:13.706806 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" event={"ID":"81fcdaba-b5bb-4265-9949-fa8a8d0a9191","Type":"ContainerDied","Data":"7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31"} Jul 24 18:26:13.707205 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:13.707188 2559 scope.go:117] "RemoveContainer" containerID="7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31" Jul 24 18:26:13.708256 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:13.708228 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:26:13.708319 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:13.708296 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:26:13.708355 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:13.708338 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:26:14.718646 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:14.718614 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" event={"ID":"81fcdaba-b5bb-4265-9949-fa8a8d0a9191","Type":"ContainerStarted","Data":"d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64"} Jul 24 18:26:15.628347 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:15.628311 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:26:15.628607 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:15.628584 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 24 18:26:15.628656 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:15.628639 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 24 18:26:16.628130 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:16.628037 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 24 18:26:16.628130 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:16.628097 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 24 18:26:17.628319 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:17.628283 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 24 18:26:17.628719 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:17.628337 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 24 18:26:18.628605 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:18.628567 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 24 18:26:18.629049 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:18.628639 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 24 18:26:19.628191 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:19.628160 2559 patch_prober.go:28] interesting pod/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" start-of-body= Jul 24 18:26:19.628371 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:19.628213 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.57:15021/healthz/ready\": dial tcp 10.132.0.57:15021: connect: connection refused" Jul 24 18:26:20.631349 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:20.631318 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:26:20.631738 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:20.631575 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:26:20.632390 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:20.632375 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:26:22.645827 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.645777 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt"] Jul 24 18:26:22.649424 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.649402 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.653046 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.653026 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d91e94\"/\"ipp-route-d91e94-kserve-self-signed-certs\"" Jul 24 18:26:22.654229 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.654209 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-d91e94\"/\"default-dockercfg-7ghrd\"" Jul 24 18:26:22.654348 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.654230 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d91e94\"/\"kube-root-ca.crt\"" Jul 24 18:26:22.654348 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.654280 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-d91e94\"/\"openshift-service-ca.crt\"" Jul 24 18:26:22.662864 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.662840 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt"] Jul 24 18:26:22.717640 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.717589 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-home\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.717853 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.717649 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-dshm\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.717853 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.717753 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-model-cache\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.717853 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.717812 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-tmp-dir\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.718040 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.717860 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3735916d-4311-486f-ac6f-6f1bd34d1220-tls-certs\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.718040 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.717893 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbskj\" (UniqueName: \"kubernetes.io/projected/3735916d-4311-486f-ac6f-6f1bd34d1220-kube-api-access-lbskj\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819160 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819125 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-home\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819160 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819162 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-dshm\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819386 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819220 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-model-cache\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819386 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819250 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-tmp-dir\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819501 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819403 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3735916d-4311-486f-ac6f-6f1bd34d1220-tls-certs\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819501 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819461 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbskj\" (UniqueName: \"kubernetes.io/projected/3735916d-4311-486f-ac6f-6f1bd34d1220-kube-api-access-lbskj\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819600 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-home\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819672 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819649 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-model-cache\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.819672 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.819665 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-tmp-dir\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.821652 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.821627 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-dshm\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.821953 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.821933 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3735916d-4311-486f-ac6f-6f1bd34d1220-tls-certs\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.828042 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.828017 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbskj\" (UniqueName: \"kubernetes.io/projected/3735916d-4311-486f-ac6f-6f1bd34d1220-kube-api-access-lbskj\") pod \"ipp-route-d91e94-kserve-866894b95f-5flwt\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:22.958456 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:22.958373 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:23.094911 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:23.094778 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt"] Jul 24 18:26:23.097278 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:26:23.097252 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3735916d_4311_486f_ac6f_6f1bd34d1220.slice/crio-b2dac5ec7fdabf8812900b6c75e2c6ba6f769092e3c5f8fec9dc00830fe1c004 WatchSource:0}: Error finding container b2dac5ec7fdabf8812900b6c75e2c6ba6f769092e3c5f8fec9dc00830fe1c004: Status 404 returned error can't find the container with id b2dac5ec7fdabf8812900b6c75e2c6ba6f769092e3c5f8fec9dc00830fe1c004 Jul 24 18:26:23.769086 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:23.769047 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" event={"ID":"3735916d-4311-486f-ac6f-6f1bd34d1220","Type":"ContainerStarted","Data":"73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f"} Jul 24 18:26:23.769086 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:23.769089 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" event={"ID":"3735916d-4311-486f-ac6f-6f1bd34d1220","Type":"ContainerStarted","Data":"b2dac5ec7fdabf8812900b6c75e2c6ba6f769092e3c5f8fec9dc00830fe1c004"} Jul 24 18:26:23.788096 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:23.788045 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" podStartSLOduration=1.7880277690000002 podStartE2EDuration="1.788027769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:26:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:26:23.786836972 +0000 UTC m=+2608.826631014" watchObservedRunningTime="2026-07-24 18:26:23.788027769 +0000 UTC m=+2608.827821809" Jul 24 18:26:25.354491 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:25.354454 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:26:32.959061 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:32.959029 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:32.959663 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:32.959107 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:32.971554 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:32.971529 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:33.827978 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:33.827945 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:26:46.558221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:46.558185 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:26:57.775602 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.775557 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2"] Jul 24 18:26:57.779431 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.779408 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.782562 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.782544 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-cleanup-1fd14b-openshift-default-dockercfg-pgg4p\"" Jul 24 18:26:57.791642 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.791619 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2"] Jul 24 18:26:57.840546 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840510 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-envoy\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.840748 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840553 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-certs\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.840748 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840577 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-token\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.840748 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840652 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-data\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.840748 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840684 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-credential-socket\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.840748 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840731 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-podinfo\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.841039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840775 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hwlf\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-kube-api-access-7hwlf\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.841039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840848 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-socket\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.841039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.840884 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fcc279-860d-4bc4-8672-dac891d53ca1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.941882 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.941853 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-envoy\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.941882 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.941883 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-certs\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942132 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.941901 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-token\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942132 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.941921 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-data\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942132 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.941944 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-credential-socket\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942132 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.941996 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-podinfo\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942132 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.942046 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hwlf\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-kube-api-access-7hwlf\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942132 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.942074 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-socket\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942132 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.942096 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fcc279-860d-4bc4-8672-dac891d53ca1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942525 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.942418 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-data\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942525 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.942440 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-certs\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942618 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.942520 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-credential-socket\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942824 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.942776 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fcc279-860d-4bc4-8672-dac891d53ca1-istiod-ca-cert\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.942911 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.942889 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-socket\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.944387 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.944366 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-envoy\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.944594 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.944576 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-podinfo\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.957946 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.957920 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-token\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:57.958111 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:57.958089 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hwlf\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-kube-api-access-7hwlf\") pod \"e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:58.089093 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:58.089061 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:58.241130 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:58.241102 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2"] Jul 24 18:26:58.243743 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:26:58.243719 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22fcc279_860d_4bc4_8672_dac891d53ca1.slice/crio-fcc1d167d7a5539ae5361209bbb96b6524dbd18ba50e0bf3d8597210346feb0b WatchSource:0}: Error finding container fcc1d167d7a5539ae5361209bbb96b6524dbd18ba50e0bf3d8597210346feb0b: Status 404 returned error can't find the container with id fcc1d167d7a5539ae5361209bbb96b6524dbd18ba50e0bf3d8597210346feb0b Jul 24 18:26:58.245918 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:58.245884 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:26:58.246004 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:58.245955 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:26:58.246004 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:58.245992 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:26:58.947025 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:58.946984 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" event={"ID":"22fcc279-860d-4bc4-8672-dac891d53ca1","Type":"ContainerStarted","Data":"33e1a5bfab09106b1a23243405466627fe2d59f02ef10635e6a72b81d521093b"} Jul 24 18:26:58.947462 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:58.947058 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" event={"ID":"22fcc279-860d-4bc4-8672-dac891d53ca1","Type":"ContainerStarted","Data":"fcc1d167d7a5539ae5361209bbb96b6524dbd18ba50e0bf3d8597210346feb0b"} Jul 24 18:26:58.970333 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:58.970281 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podStartSLOduration=1.970264391 podStartE2EDuration="1.970264391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:26:58.969101285 +0000 UTC m=+2644.008895328" watchObservedRunningTime="2026-07-24 18:26:58.970264391 +0000 UTC m=+2644.010058497" Jul 24 18:26:59.090325 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:59.090291 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:59.095537 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:59.095510 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:59.952466 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:59.952430 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:26:59.953631 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:26:59.953591 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:27:07.656092 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:07.656058 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:27:15.994032 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:15.993984 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Jul 24 18:27:15.994471 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:15.994080 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Jul 24 18:27:16.503765 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.503730 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr"] Jul 24 18:27:16.507326 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.507309 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" Jul 24 18:27:16.509980 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.509954 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:27:16.510102 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.509981 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:27:16.515310 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.515096 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr"] Jul 24 18:27:16.622727 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.622687 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c738c1f-14bf-4b9b-a7d8-3df4dadece7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr\" (UID: \"3c738c1f-14bf-4b9b-a7d8-3df4dadece7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" Jul 24 18:27:16.723506 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.723453 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c738c1f-14bf-4b9b-a7d8-3df4dadece7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr\" (UID: \"3c738c1f-14bf-4b9b-a7d8-3df4dadece7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" Jul 24 18:27:16.724161 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.724139 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c738c1f-14bf-4b9b-a7d8-3df4dadece7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr\" (UID: \"3c738c1f-14bf-4b9b-a7d8-3df4dadece7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" Jul 24 18:27:16.820323 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:16.820287 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" Jul 24 18:27:17.011726 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:17.011701 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr"] Jul 24 18:27:17.013763 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:27:17.013732 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c738c1f_14bf_4b9b_a7d8_3df4dadece7b.slice/crio-ee482d0519ffd00a51f6ac28523a35fb226422d2fa2c93151e1903b2d956603d WatchSource:0}: Error finding container ee482d0519ffd00a51f6ac28523a35fb226422d2fa2c93151e1903b2d956603d: Status 404 returned error can't find the container with id ee482d0519ffd00a51f6ac28523a35fb226422d2fa2c93151e1903b2d956603d Jul 24 18:27:17.042541 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:17.042514 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" event={"ID":"3c738c1f-14bf-4b9b-a7d8-3df4dadece7b","Type":"ContainerStarted","Data":"ee482d0519ffd00a51f6ac28523a35fb226422d2fa2c93151e1903b2d956603d"} Jul 24 18:27:18.054726 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:18.054633 2559 generic.go:356] "Generic (PLEG): container finished" podID="3c738c1f-14bf-4b9b-a7d8-3df4dadece7b" containerID="fbcc31bf17acbab88eea5927646d370e62db100b241d8b088709bc3270319c15" exitCode=0 Jul 24 18:27:18.055243 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:18.054733 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" event={"ID":"3c738c1f-14bf-4b9b-a7d8-3df4dadece7b","Type":"ContainerDied","Data":"fbcc31bf17acbab88eea5927646d370e62db100b241d8b088709bc3270319c15"} Jul 24 18:27:18.056437 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:18.056415 2559 generic.go:356] "Generic (PLEG): container finished" podID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerID="33e1a5bfab09106b1a23243405466627fe2d59f02ef10635e6a72b81d521093b" exitCode=0 Jul 24 18:27:18.056556 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:18.056488 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" event={"ID":"22fcc279-860d-4bc4-8672-dac891d53ca1","Type":"ContainerDied","Data":"33e1a5bfab09106b1a23243405466627fe2d59f02ef10635e6a72b81d521093b"} Jul 24 18:27:18.056864 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:18.056849 2559 scope.go:117] "RemoveContainer" containerID="33e1a5bfab09106b1a23243405466627fe2d59f02ef10635e6a72b81d521093b" Jul 24 18:27:18.057896 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:18.057866 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:27:18.057964 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:18.057948 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:27:18.058002 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:18.057985 2559 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892156Ki","pods":"250"} Jul 24 18:27:19.063984 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:19.063949 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" event={"ID":"22fcc279-860d-4bc4-8672-dac891d53ca1","Type":"ContainerStarted","Data":"1d3332cd8ce7bdecabee51bd0c682093268a67b69b1d15441033fed8e1311ce9"} Jul 24 18:27:19.090255 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:19.090223 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:27:19.090633 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:19.090543 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 18:27:19.090832 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:19.090778 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 18:27:19.238523 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:19.238499 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" Jul 24 18:27:19.348196 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:19.348105 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c738c1f-14bf-4b9b-a7d8-3df4dadece7b-maas-api-service-ca\") pod \"3c738c1f-14bf-4b9b-a7d8-3df4dadece7b\" (UID: \"3c738c1f-14bf-4b9b-a7d8-3df4dadece7b\") " Jul 24 18:27:19.348453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:19.348429 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c738c1f-14bf-4b9b-a7d8-3df4dadece7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3c738c1f-14bf-4b9b-a7d8-3df4dadece7b" (UID: "3c738c1f-14bf-4b9b-a7d8-3df4dadece7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:27:19.449198 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:19.449167 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c738c1f-14bf-4b9b-a7d8-3df4dadece7b-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:20.070126 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:20.070095 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" Jul 24 18:27:20.070539 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:20.070172 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr" event={"ID":"3c738c1f-14bf-4b9b-a7d8-3df4dadece7b","Type":"ContainerDied","Data":"ee482d0519ffd00a51f6ac28523a35fb226422d2fa2c93151e1903b2d956603d"} Jul 24 18:27:20.070539 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:20.070219 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee482d0519ffd00a51f6ac28523a35fb226422d2fa2c93151e1903b2d956603d" Jul 24 18:27:20.090150 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:20.090115 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 18:27:20.090338 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:20.090181 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 18:27:21.089701 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:21.089667 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 18:27:21.090358 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:21.090307 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 18:27:22.090323 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:22.090279 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 18:27:22.090856 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:22.090347 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 18:27:23.089669 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:23.089632 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 18:27:23.090007 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:23.089974 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 18:27:24.091846 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:24.091801 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" start-of-body= Jul 24 18:27:24.092370 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:24.091871 2559 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.59:15021/healthz/ready\": dial tcp 10.132.0.59:15021: connect: connection refused" Jul 24 18:27:25.093322 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:25.093288 2559 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:27:25.093810 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:25.093559 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:27:25.094360 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:25.094343 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:27:30.055177 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:30.055138 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:27:31.596088 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:31.596055 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr"] Jul 24 18:27:31.599015 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:31.598987 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-1fd14b-grmrr"] Jul 24 18:27:33.494144 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:33.494108 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c738c1f-14bf-4b9b-a7d8-3df4dadece7b" path="/var/lib/kubelet/pods/3c738c1f-14bf-4b9b-a7d8-3df4dadece7b/volumes" Jul 24 18:27:37.951730 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:37.951694 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2"] Jul 24 18:27:37.952156 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:37.951982 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" containerID="cri-o://1d3332cd8ce7bdecabee51bd0c682093268a67b69b1d15441033fed8e1311ce9" gracePeriod=30 Jul 24 18:27:40.095691 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:40.095658 2559 patch_prober.go:28] interesting pod/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 24 18:27:40.096145 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:40.095719 2559 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 24 18:27:42.057717 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.057685 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh"] Jul 24 18:27:42.058130 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.058069 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c738c1f-14bf-4b9b-a7d8-3df4dadece7b" containerName="revoke-keys" Jul 24 18:27:42.058130 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.058081 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c738c1f-14bf-4b9b-a7d8-3df4dadece7b" containerName="revoke-keys" Jul 24 18:27:42.058209 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.058155 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c738c1f-14bf-4b9b-a7d8-3df4dadece7b" containerName="revoke-keys" Jul 24 18:27:42.061195 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.061178 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" Jul 24 18:27:42.064078 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.064062 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:27:42.064173 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.064069 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:27:42.074453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.074423 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh"] Jul 24 18:27:42.145702 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.145659 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7655e047-bd52-4bca-b659-0e5e3b79f8d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh\" (UID: \"7655e047-bd52-4bca-b659-0e5e3b79f8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" Jul 24 18:27:42.246873 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.246829 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7655e047-bd52-4bca-b659-0e5e3b79f8d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh\" (UID: \"7655e047-bd52-4bca-b659-0e5e3b79f8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" Jul 24 18:27:42.247486 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.247463 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7655e047-bd52-4bca-b659-0e5e3b79f8d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh\" (UID: \"7655e047-bd52-4bca-b659-0e5e3b79f8d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" Jul 24 18:27:42.381605 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.381563 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" Jul 24 18:27:42.521892 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:42.521867 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh"] Jul 24 18:27:42.524293 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:27:42.524249 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7655e047_bd52_4bca_b659_0e5e3b79f8d1.slice/crio-6b9188e84cfa38e3bb27d0a4c0265bbc5b95852e709b2402da954149967c6a99 WatchSource:0}: Error finding container 6b9188e84cfa38e3bb27d0a4c0265bbc5b95852e709b2402da954149967c6a99: Status 404 returned error can't find the container with id 6b9188e84cfa38e3bb27d0a4c0265bbc5b95852e709b2402da954149967c6a99 Jul 24 18:27:43.193949 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.193915 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" event={"ID":"7655e047-bd52-4bca-b659-0e5e3b79f8d1","Type":"ContainerStarted","Data":"8343af8b1e7aa3d9ee632b9098db243f164a905f38c31afbb692947f64e697fe"} Jul 24 18:27:43.194356 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.193980 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" event={"ID":"7655e047-bd52-4bca-b659-0e5e3b79f8d1","Type":"ContainerStarted","Data":"6b9188e84cfa38e3bb27d0a4c0265bbc5b95852e709b2402da954149967c6a99"} Jul 24 18:27:43.196059 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.196033 2559 generic.go:356] "Generic (PLEG): container finished" podID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerID="1d3332cd8ce7bdecabee51bd0c682093268a67b69b1d15441033fed8e1311ce9" exitCode=0 Jul 24 18:27:43.196172 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.196099 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" event={"ID":"22fcc279-860d-4bc4-8672-dac891d53ca1","Type":"ContainerDied","Data":"1d3332cd8ce7bdecabee51bd0c682093268a67b69b1d15441033fed8e1311ce9"} Jul 24 18:27:43.196172 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.196158 2559 scope.go:117] "RemoveContainer" containerID="33e1a5bfab09106b1a23243405466627fe2d59f02ef10635e6a72b81d521093b" Jul 24 18:27:43.212743 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.212655 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" podStartSLOduration=1.212635867 podStartE2EDuration="1.212635867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:27:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:27:43.210148396 +0000 UTC m=+2688.249942438" watchObservedRunningTime="2026-07-24 18:27:43.212635867 +0000 UTC m=+2688.252429907" Jul 24 18:27:43.263728 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.263704 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:27:43.357490 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357455 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-certs\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357672 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357512 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hwlf\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-kube-api-access-7hwlf\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357672 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357548 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-data\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357672 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357572 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-token\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357672 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357590 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-socket\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357672 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357653 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-credential-socket\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357908 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357686 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fcc279-860d-4bc4-8672-dac891d53ca1-istiod-ca-cert\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357908 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357709 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:27:43.357908 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357730 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-podinfo\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357908 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357756 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-envoy\") pod \"22fcc279-860d-4bc4-8672-dac891d53ca1\" (UID: \"22fcc279-860d-4bc4-8672-dac891d53ca1\") " Jul 24 18:27:43.357908 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.357865 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-data" (OuterVolumeSpecName: "istio-data") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:27:43.358147 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.358125 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22fcc279-860d-4bc4-8672-dac891d53ca1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:27:43.358224 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.358182 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:43.358224 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.358199 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-data\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:43.358377 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.358352 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:27:43.358635 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.358555 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:27:43.359995 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.359971 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-kube-api-access-7hwlf" (OuterVolumeSpecName: "kube-api-access-7hwlf") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "kube-api-access-7hwlf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:27:43.360098 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.360018 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:27:43.360098 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.360041 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:27:43.360098 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.360095 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-token" (OuterVolumeSpecName: "istio-token") pod "22fcc279-860d-4bc4-8672-dac891d53ca1" (UID: "22fcc279-860d-4bc4-8672-dac891d53ca1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:27:43.458966 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.458937 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hwlf\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-kube-api-access-7hwlf\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:43.458966 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.458964 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-token\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:43.458966 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.458973 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-workload-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:43.459205 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.458983 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-credential-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:43.459205 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.458991 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22fcc279-860d-4bc4-8672-dac891d53ca1-istiod-ca-cert\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:43.459205 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.458998 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-podinfo\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:43.459205 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:43.459005 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22fcc279-860d-4bc4-8672-dac891d53ca1-istio-envoy\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:44.202544 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:44.202505 2559 generic.go:356] "Generic (PLEG): container finished" podID="7655e047-bd52-4bca-b659-0e5e3b79f8d1" containerID="8343af8b1e7aa3d9ee632b9098db243f164a905f38c31afbb692947f64e697fe" exitCode=0 Jul 24 18:27:44.202962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:44.202592 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" event={"ID":"7655e047-bd52-4bca-b659-0e5e3b79f8d1","Type":"ContainerDied","Data":"8343af8b1e7aa3d9ee632b9098db243f164a905f38c31afbb692947f64e697fe"} Jul 24 18:27:44.203835 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:44.203814 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" Jul 24 18:27:44.203954 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:44.203819 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2" event={"ID":"22fcc279-860d-4bc4-8672-dac891d53ca1","Type":"ContainerDied","Data":"fcc1d167d7a5539ae5361209bbb96b6524dbd18ba50e0bf3d8597210346feb0b"} Jul 24 18:27:44.203996 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:44.203978 2559 scope.go:117] "RemoveContainer" containerID="1d3332cd8ce7bdecabee51bd0c682093268a67b69b1d15441033fed8e1311ce9" Jul 24 18:27:44.234031 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:44.233999 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2"] Jul 24 18:27:44.238170 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:44.238145 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-cleanup-1fd14b-openshift-default-5d845fbfc6-chhn2"] Jul 24 18:27:45.348208 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:45.348177 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" Jul 24 18:27:45.480641 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:45.480555 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7655e047-bd52-4bca-b659-0e5e3b79f8d1-maas-api-service-ca\") pod \"7655e047-bd52-4bca-b659-0e5e3b79f8d1\" (UID: \"7655e047-bd52-4bca-b659-0e5e3b79f8d1\") " Jul 24 18:27:45.480943 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:45.480922 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7655e047-bd52-4bca-b659-0e5e3b79f8d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7655e047-bd52-4bca-b659-0e5e3b79f8d1" (UID: "7655e047-bd52-4bca-b659-0e5e3b79f8d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:27:45.495211 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:45.495177 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" path="/var/lib/kubelet/pods/22fcc279-860d-4bc4-8672-dac891d53ca1/volumes" Jul 24 18:27:45.582250 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:45.582215 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7655e047-bd52-4bca-b659-0e5e3b79f8d1-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:27:46.221451 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:46.221425 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" Jul 24 18:27:46.221451 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:46.221438 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh" event={"ID":"7655e047-bd52-4bca-b659-0e5e3b79f8d1","Type":"ContainerDied","Data":"6b9188e84cfa38e3bb27d0a4c0265bbc5b95852e709b2402da954149967c6a99"} Jul 24 18:27:46.221703 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:46.221502 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b9188e84cfa38e3bb27d0a4c0265bbc5b95852e709b2402da954149967c6a99" Jul 24 18:27:55.557016 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:55.556984 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:27:55.644916 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:55.644884 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:27:55.650091 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:27:55.650062 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:28:02.145058 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:02.145025 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh"] Jul 24 18:28:02.147498 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:02.147475 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-d91e94-fbjmh"] Jul 24 18:28:03.493864 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:03.493824 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7655e047-bd52-4bca-b659-0e5e3b79f8d1" path="/var/lib/kubelet/pods/7655e047-bd52-4bca-b659-0e5e3b79f8d1/volumes" Jul 24 18:28:04.561742 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:04.561709 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:28:07.540336 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:07.540303 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt"] Jul 24 18:28:22.560194 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.560147 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" podUID="3735916d-4311-486f-ac6f-6f1bd34d1220" containerName="main" containerID="cri-o://73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f" gracePeriod=45 Jul 24 18:28:22.810413 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.810355 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:28:22.908063 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908024 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-tmp-dir\") pod \"3735916d-4311-486f-ac6f-6f1bd34d1220\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " Jul 24 18:28:22.908063 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908062 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-model-cache\") pod \"3735916d-4311-486f-ac6f-6f1bd34d1220\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " Jul 24 18:28:22.908314 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908089 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-home\") pod \"3735916d-4311-486f-ac6f-6f1bd34d1220\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " Jul 24 18:28:22.908314 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908116 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-dshm\") pod \"3735916d-4311-486f-ac6f-6f1bd34d1220\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " Jul 24 18:28:22.908314 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908163 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3735916d-4311-486f-ac6f-6f1bd34d1220-tls-certs\") pod \"3735916d-4311-486f-ac6f-6f1bd34d1220\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " Jul 24 18:28:22.908314 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908186 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lbskj\" (UniqueName: \"kubernetes.io/projected/3735916d-4311-486f-ac6f-6f1bd34d1220-kube-api-access-lbskj\") pod \"3735916d-4311-486f-ac6f-6f1bd34d1220\" (UID: \"3735916d-4311-486f-ac6f-6f1bd34d1220\") " Jul 24 18:28:22.908493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908368 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3735916d-4311-486f-ac6f-6f1bd34d1220" (UID: "3735916d-4311-486f-ac6f-6f1bd34d1220"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:22.908493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908356 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-model-cache" (OuterVolumeSpecName: "model-cache") pod "3735916d-4311-486f-ac6f-6f1bd34d1220" (UID: "3735916d-4311-486f-ac6f-6f1bd34d1220"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:22.908493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908408 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-home" (OuterVolumeSpecName: "home") pod "3735916d-4311-486f-ac6f-6f1bd34d1220" (UID: "3735916d-4311-486f-ac6f-6f1bd34d1220"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:22.908641 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908566 2559 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-tmp-dir\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:22.908641 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908582 2559 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-model-cache\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:22.908641 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.908590 2559 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-home\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:22.910534 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.910509 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-dshm" (OuterVolumeSpecName: "dshm") pod "3735916d-4311-486f-ac6f-6f1bd34d1220" (UID: "3735916d-4311-486f-ac6f-6f1bd34d1220"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:22.910656 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.910546 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3735916d-4311-486f-ac6f-6f1bd34d1220-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3735916d-4311-486f-ac6f-6f1bd34d1220" (UID: "3735916d-4311-486f-ac6f-6f1bd34d1220"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 24 18:28:22.910656 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:22.910565 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3735916d-4311-486f-ac6f-6f1bd34d1220-kube-api-access-lbskj" (OuterVolumeSpecName: "kube-api-access-lbskj") pod "3735916d-4311-486f-ac6f-6f1bd34d1220" (UID: "3735916d-4311-486f-ac6f-6f1bd34d1220"). InnerVolumeSpecName "kube-api-access-lbskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:28:23.009022 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.008985 2559 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3735916d-4311-486f-ac6f-6f1bd34d1220-dshm\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:23.009022 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.009010 2559 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3735916d-4311-486f-ac6f-6f1bd34d1220-tls-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:23.009022 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.009020 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbskj\" (UniqueName: \"kubernetes.io/projected/3735916d-4311-486f-ac6f-6f1bd34d1220-kube-api-access-lbskj\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:23.405082 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.405050 2559 generic.go:356] "Generic (PLEG): container finished" podID="3735916d-4311-486f-ac6f-6f1bd34d1220" containerID="73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f" exitCode=0 Jul 24 18:28:23.405251 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.405094 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" Jul 24 18:28:23.405251 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.405129 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" event={"ID":"3735916d-4311-486f-ac6f-6f1bd34d1220","Type":"ContainerDied","Data":"73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f"} Jul 24 18:28:23.405251 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.405174 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt" event={"ID":"3735916d-4311-486f-ac6f-6f1bd34d1220","Type":"ContainerDied","Data":"b2dac5ec7fdabf8812900b6c75e2c6ba6f769092e3c5f8fec9dc00830fe1c004"} Jul 24 18:28:23.405251 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.405192 2559 scope.go:117] "RemoveContainer" containerID="73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f" Jul 24 18:28:23.414280 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.414264 2559 scope.go:117] "RemoveContainer" containerID="73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f" Jul 24 18:28:23.414532 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:28:23.414514 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f\": container with ID starting with 73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f not found: ID does not exist" containerID="73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f" Jul 24 18:28:23.414579 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.414541 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f"} err="failed to get container status \"73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f\": rpc error: code = NotFound desc = could not find container \"73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f\": container with ID starting with 73a10726287782b21822c733dad0ac00327ac51f06b14b98a3730f62616d861f not found: ID does not exist" Jul 24 18:28:23.429772 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.429747 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt"] Jul 24 18:28:23.435526 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.435501 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-d91e94/ipp-route-d91e94-kserve-866894b95f-5flwt"] Jul 24 18:28:23.494448 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:23.494418 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3735916d-4311-486f-ac6f-6f1bd34d1220" path="/var/lib/kubelet/pods/3735916d-4311-486f-ac6f-6f1bd34d1220/volumes" Jul 24 18:28:50.636949 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:50.636915 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh"] Jul 24 18:28:50.637427 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:50.637155 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" containerID="cri-o://c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675" gracePeriod=30 Jul 24 18:28:51.011689 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.011607 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx"] Jul 24 18:28:51.012056 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012040 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7655e047-bd52-4bca-b659-0e5e3b79f8d1" containerName="revoke-keys" Jul 24 18:28:51.012112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012058 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="7655e047-bd52-4bca-b659-0e5e3b79f8d1" containerName="revoke-keys" Jul 24 18:28:51.012112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012069 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" Jul 24 18:28:51.012112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012074 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" Jul 24 18:28:51.012112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012081 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" Jul 24 18:28:51.012112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012090 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" Jul 24 18:28:51.012112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012103 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3735916d-4311-486f-ac6f-6f1bd34d1220" containerName="main" Jul 24 18:28:51.012112 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012109 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3735916d-4311-486f-ac6f-6f1bd34d1220" containerName="main" Jul 24 18:28:51.012346 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012166 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" Jul 24 18:28:51.012346 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012181 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3735916d-4311-486f-ac6f-6f1bd34d1220" containerName="main" Jul 24 18:28:51.012346 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.012188 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="7655e047-bd52-4bca-b659-0e5e3b79f8d1" containerName="revoke-keys" Jul 24 18:28:51.015219 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.015202 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" Jul 24 18:28:51.020495 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.020466 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:28:51.020650 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.020606 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:28:51.030764 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.030737 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx"] Jul 24 18:28:51.160893 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.160861 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169e3b08-577a-4ed5-a63f-d6d49df35a84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx\" (UID: \"169e3b08-577a-4ed5-a63f-d6d49df35a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" Jul 24 18:28:51.262332 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.262243 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169e3b08-577a-4ed5-a63f-d6d49df35a84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx\" (UID: \"169e3b08-577a-4ed5-a63f-d6d49df35a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" Jul 24 18:28:51.262874 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.262853 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169e3b08-577a-4ed5-a63f-d6d49df35a84-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx\" (UID: \"169e3b08-577a-4ed5-a63f-d6d49df35a84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" Jul 24 18:28:51.325409 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.325376 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" Jul 24 18:28:51.460117 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.460094 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx"] Jul 24 18:28:51.462084 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:28:51.462054 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169e3b08_577a_4ed5_a63f_d6d49df35a84.slice/crio-899eb9d084b61c44eb9231352f366eafe36716087804c1ed8f5fa8e356d3aadf WatchSource:0}: Error finding container 899eb9d084b61c44eb9231352f366eafe36716087804c1ed8f5fa8e356d3aadf: Status 404 returned error can't find the container with id 899eb9d084b61c44eb9231352f366eafe36716087804c1ed8f5fa8e356d3aadf Jul 24 18:28:51.540989 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:51.540904 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" event={"ID":"169e3b08-577a-4ed5-a63f-d6d49df35a84","Type":"ContainerStarted","Data":"899eb9d084b61c44eb9231352f366eafe36716087804c1ed8f5fa8e356d3aadf"} Jul 24 18:28:52.547745 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:52.547652 2559 generic.go:356] "Generic (PLEG): container finished" podID="169e3b08-577a-4ed5-a63f-d6d49df35a84" containerID="5ea877719cbab61d64c85e86ffae7d9eecfbd24344f124344b6ce42ff6429469" exitCode=0 Jul 24 18:28:52.548150 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:52.547742 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" event={"ID":"169e3b08-577a-4ed5-a63f-d6d49df35a84","Type":"ContainerDied","Data":"5ea877719cbab61d64c85e86ffae7d9eecfbd24344f124344b6ce42ff6429469"} Jul 24 18:28:53.687113 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:53.687085 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" Jul 24 18:28:53.784730 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:53.784693 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169e3b08-577a-4ed5-a63f-d6d49df35a84-maas-api-service-ca\") pod \"169e3b08-577a-4ed5-a63f-d6d49df35a84\" (UID: \"169e3b08-577a-4ed5-a63f-d6d49df35a84\") " Jul 24 18:28:53.785068 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:53.785045 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/169e3b08-577a-4ed5-a63f-d6d49df35a84-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "169e3b08-577a-4ed5-a63f-d6d49df35a84" (UID: "169e3b08-577a-4ed5-a63f-d6d49df35a84"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:28:53.886157 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:53.886127 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/169e3b08-577a-4ed5-a63f-d6d49df35a84-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:54.558519 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:54.558482 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" Jul 24 18:28:54.558519 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:54.558512 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx" event={"ID":"169e3b08-577a-4ed5-a63f-d6d49df35a84","Type":"ContainerDied","Data":"899eb9d084b61c44eb9231352f366eafe36716087804c1ed8f5fa8e356d3aadf"} Jul 24 18:28:54.558746 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:54.558567 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="899eb9d084b61c44eb9231352f366eafe36716087804c1ed8f5fa8e356d3aadf" Jul 24 18:28:55.896039 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:55.896017 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:28:56.003814 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.003710 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24dkt\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-kube-api-access-24dkt\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.003814 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.003759 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e18dfc-e25b-47b7-b493-a917cadd743c-istiod-ca-cert\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.003814 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.003811 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-envoy\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.004078 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.003834 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-data\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.004078 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.003988 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-credential-socket\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.004078 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004033 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-token\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.004230 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004080 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-certs\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.004230 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004105 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-socket\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.004230 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004135 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-data" (OuterVolumeSpecName: "istio-data") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:56.004390 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004250 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-podinfo\") pod \"79e18dfc-e25b-47b7-b493-a917cadd743c\" (UID: \"79e18dfc-e25b-47b7-b493-a917cadd743c\") " Jul 24 18:28:56.004390 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004261 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:56.004390 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004280 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79e18dfc-e25b-47b7-b493-a917cadd743c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:28:56.004390 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004310 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:56.004616 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004471 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:56.004616 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004573 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.004616 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004596 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/79e18dfc-e25b-47b7-b493-a917cadd743c-istiod-ca-cert\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.004616 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004612 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-data\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.005020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004624 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-credential-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.005020 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.004636 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-workload-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.006284 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.006250 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-kube-api-access-24dkt" (OuterVolumeSpecName: "kube-api-access-24dkt") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "kube-api-access-24dkt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:28:56.006284 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.006270 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-token" (OuterVolumeSpecName: "istio-token") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:28:56.006439 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.006354 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:28:56.006439 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.006382 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "79e18dfc-e25b-47b7-b493-a917cadd743c" (UID: "79e18dfc-e25b-47b7-b493-a917cadd743c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:28:56.105241 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.105209 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24dkt\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-kube-api-access-24dkt\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.105241 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.105239 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-envoy\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.105241 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.105250 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-token\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.105483 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.105262 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/79e18dfc-e25b-47b7-b493-a917cadd743c-istio-podinfo\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:28:56.570311 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.570277 2559 generic.go:356] "Generic (PLEG): container finished" podID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerID="c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675" exitCode=0 Jul 24 18:28:56.570582 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.570332 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" Jul 24 18:28:56.570582 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.570349 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" event={"ID":"79e18dfc-e25b-47b7-b493-a917cadd743c","Type":"ContainerDied","Data":"c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675"} Jul 24 18:28:56.570582 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.570391 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh" event={"ID":"79e18dfc-e25b-47b7-b493-a917cadd743c","Type":"ContainerDied","Data":"c05837af70b670d85955fd3a12cd74b097e6a070c95a59019e5fd06d732ef08a"} Jul 24 18:28:56.570582 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.570412 2559 scope.go:117] "RemoveContainer" containerID="c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675" Jul 24 18:28:56.579752 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.579735 2559 scope.go:117] "RemoveContainer" containerID="352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6" Jul 24 18:28:56.587975 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.587955 2559 scope.go:117] "RemoveContainer" containerID="c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675" Jul 24 18:28:56.588246 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:28:56.588225 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675\": container with ID starting with c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675 not found: ID does not exist" containerID="c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675" Jul 24 18:28:56.588310 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.588255 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675"} err="failed to get container status \"c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675\": rpc error: code = NotFound desc = could not find container \"c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675\": container with ID starting with c9bcd8c8eab60cd521331c2e2198f83e8d0fa7d1675f6468e315d9a173fc2675 not found: ID does not exist" Jul 24 18:28:56.588310 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.588271 2559 scope.go:117] "RemoveContainer" containerID="352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6" Jul 24 18:28:56.588496 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:28:56.588481 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6\": container with ID starting with 352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6 not found: ID does not exist" containerID="352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6" Jul 24 18:28:56.588539 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.588502 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6"} err="failed to get container status \"352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6\": rpc error: code = NotFound desc = could not find container \"352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6\": container with ID starting with 352f8d94d9a1157067461eecdac55313ccffdabf633aa7e394634ff82d95eaf6 not found: ID does not exist" Jul 24 18:28:56.593220 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.593200 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh"] Jul 24 18:28:56.597974 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:56.597954 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-d91e94-openshift-default-5dd8dfdcf4-57qvh"] Jul 24 18:28:57.494247 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:28:57.494214 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" path="/var/lib/kubelet/pods/79e18dfc-e25b-47b7-b493-a917cadd743c/volumes" Jul 24 18:29:04.075216 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:04.075179 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:29:06.077874 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:06.077839 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx"] Jul 24 18:29:06.080283 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:06.080257 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-651f67-9tzbx"] Jul 24 18:29:07.494235 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:07.494199 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169e3b08-577a-4ed5-a63f-d6d49df35a84" path="/var/lib/kubelet/pods/169e3b08-577a-4ed5-a63f-d6d49df35a84/volumes" Jul 24 18:29:12.671932 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:12.671899 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts"] Jul 24 18:29:12.672409 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:12.672101 2559 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" containerID="cri-o://d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64" gracePeriod=30 Jul 24 18:29:13.063407 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063380 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h"] Jul 24 18:29:13.063715 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063704 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" Jul 24 18:29:13.063760 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063718 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" Jul 24 18:29:13.063760 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063740 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169e3b08-577a-4ed5-a63f-d6d49df35a84" containerName="revoke-keys" Jul 24 18:29:13.063760 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063747 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="169e3b08-577a-4ed5-a63f-d6d49df35a84" containerName="revoke-keys" Jul 24 18:29:13.063872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063823 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="22fcc279-860d-4bc4-8672-dac891d53ca1" containerName="istio-proxy" Jul 24 18:29:13.063872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063838 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" Jul 24 18:29:13.063872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063848 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" Jul 24 18:29:13.063872 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.063858 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="169e3b08-577a-4ed5-a63f-d6d49df35a84" containerName="revoke-keys" Jul 24 18:29:13.067102 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.067087 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" Jul 24 18:29:13.069708 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.069690 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:29:13.069775 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.069716 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:29:13.073489 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.073448 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h"] Jul 24 18:29:13.150279 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.150241 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428f039c-1908-42b0-8ada-36d25dc0c6f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h\" (UID: \"428f039c-1908-42b0-8ada-36d25dc0c6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" Jul 24 18:29:13.251001 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.250965 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428f039c-1908-42b0-8ada-36d25dc0c6f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h\" (UID: \"428f039c-1908-42b0-8ada-36d25dc0c6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" Jul 24 18:29:13.251548 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.251528 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428f039c-1908-42b0-8ada-36d25dc0c6f7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h\" (UID: \"428f039c-1908-42b0-8ada-36d25dc0c6f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" Jul 24 18:29:13.379156 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.379070 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" Jul 24 18:29:13.521301 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.521276 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h"] Jul 24 18:29:13.651889 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:13.651804 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" event={"ID":"428f039c-1908-42b0-8ada-36d25dc0c6f7","Type":"ContainerStarted","Data":"6e59b2c440e5f08cc63da2f82dad90e82ba4994c12e0f2dcfec2d109f0b71596"} Jul 24 18:29:14.658923 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:14.658888 2559 generic.go:356] "Generic (PLEG): container finished" podID="428f039c-1908-42b0-8ada-36d25dc0c6f7" containerID="5d05b8541b465bc760f19d1e1e02179aaf7b65983e39a4882a5ea201d6998911" exitCode=0 Jul 24 18:29:14.659321 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:14.658938 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" event={"ID":"428f039c-1908-42b0-8ada-36d25dc0c6f7","Type":"ContainerDied","Data":"5d05b8541b465bc760f19d1e1e02179aaf7b65983e39a4882a5ea201d6998911"} Jul 24 18:29:15.795065 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:15.795040 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" Jul 24 18:29:15.877231 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:15.877200 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428f039c-1908-42b0-8ada-36d25dc0c6f7-maas-api-service-ca\") pod \"428f039c-1908-42b0-8ada-36d25dc0c6f7\" (UID: \"428f039c-1908-42b0-8ada-36d25dc0c6f7\") " Jul 24 18:29:15.877565 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:15.877543 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/428f039c-1908-42b0-8ada-36d25dc0c6f7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "428f039c-1908-42b0-8ada-36d25dc0c6f7" (UID: "428f039c-1908-42b0-8ada-36d25dc0c6f7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:29:15.978672 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:15.978585 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/428f039c-1908-42b0-8ada-36d25dc0c6f7-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:16.669177 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:16.669095 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" event={"ID":"428f039c-1908-42b0-8ada-36d25dc0c6f7","Type":"ContainerDied","Data":"6e59b2c440e5f08cc63da2f82dad90e82ba4994c12e0f2dcfec2d109f0b71596"} Jul 24 18:29:16.669177 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:16.669131 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h" Jul 24 18:29:16.669177 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:16.669153 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e59b2c440e5f08cc63da2f82dad90e82ba4994c12e0f2dcfec2d109f0b71596" Jul 24 18:29:17.923803 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.923767 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:29:17.998131 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998038 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istiod-ca-cert\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998131 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998085 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-949n8\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-kube-api-access-949n8\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998131 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998105 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-certs\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998131 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998126 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-envoy\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998480 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998298 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-data\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998480 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998354 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-socket\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998480 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998381 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:29:17.998480 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998443 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-credential-socket\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998497 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:29:17.998684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998509 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-token\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998533 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-data" (OuterVolumeSpecName: "istio-data") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:29:17.998684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998554 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-podinfo\") pod \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\" (UID: \"81fcdaba-b5bb-4265-9949-fa8a8d0a9191\") " Jul 24 18:29:17.998684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998640 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:29:17.998684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998675 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:29:17.999021 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998923 2559 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-credential-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:17.999021 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.998945 2559 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istiod-ca-cert\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:17.999284 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.999041 2559 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-certs\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:17.999284 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.999134 2559 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-data\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:17.999284 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:17.999146 2559 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-workload-socket\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:18.000561 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.000533 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 24 18:29:18.000561 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.000539 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-kube-api-access-949n8" (OuterVolumeSpecName: "kube-api-access-949n8") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "kube-api-access-949n8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:29:18.000732 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.000567 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-token" (OuterVolumeSpecName: "istio-token") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 24 18:29:18.000811 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.000736 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "81fcdaba-b5bb-4265-9949-fa8a8d0a9191" (UID: "81fcdaba-b5bb-4265-9949-fa8a8d0a9191"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 24 18:29:18.099587 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.099551 2559 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-token\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:18.099587 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.099578 2559 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-podinfo\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:18.099587 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.099590 2559 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-949n8\" (UniqueName: \"kubernetes.io/projected/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-kube-api-access-949n8\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:18.099587 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.099600 2559 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fcdaba-b5bb-4265-9949-fa8a8d0a9191-istio-envoy\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:18.681334 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.681298 2559 generic.go:356] "Generic (PLEG): container finished" podID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerID="d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64" exitCode=0 Jul 24 18:29:18.681491 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.681357 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" Jul 24 18:29:18.681491 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.681384 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" event={"ID":"81fcdaba-b5bb-4265-9949-fa8a8d0a9191","Type":"ContainerDied","Data":"d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64"} Jul 24 18:29:18.681491 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.681441 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts" event={"ID":"81fcdaba-b5bb-4265-9949-fa8a8d0a9191","Type":"ContainerDied","Data":"90d5a2bcbd1dbb9440319bb058b0848a3421abb161363230cea447f7f0895d0d"} Jul 24 18:29:18.681491 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.681463 2559 scope.go:117] "RemoveContainer" containerID="d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64" Jul 24 18:29:18.691539 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.691524 2559 scope.go:117] "RemoveContainer" containerID="7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31" Jul 24 18:29:18.699573 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.699556 2559 scope.go:117] "RemoveContainer" containerID="d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64" Jul 24 18:29:18.699830 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:29:18.699807 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64\": container with ID starting with d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64 not found: ID does not exist" containerID="d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64" Jul 24 18:29:18.699894 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.699839 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64"} err="failed to get container status \"d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64\": rpc error: code = NotFound desc = could not find container \"d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64\": container with ID starting with d58a29cd750f72c289640ae9922fdcd64a62fcff2a60a315583fa94127ee7b64 not found: ID does not exist" Jul 24 18:29:18.699894 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.699857 2559 scope.go:117] "RemoveContainer" containerID="7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31" Jul 24 18:29:18.700089 ip-10-0-141-191 kubenswrapper[2559]: E0724 18:29:18.700072 2559 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31\": container with ID starting with 7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31 not found: ID does not exist" containerID="7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31" Jul 24 18:29:18.700131 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.700094 2559 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31"} err="failed to get container status \"7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31\": rpc error: code = NotFound desc = could not find container \"7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31\": container with ID starting with 7227b6ee2b3bea1e9bf15f2a9ca5426f46de669f4aa3a37ae884f047af7b4f31 not found: ID does not exist" Jul 24 18:29:18.733932 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.733886 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts"] Jul 24 18:29:18.750870 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:18.750845 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-651f67-openshift-default-7f89b4bbf8-68pts"] Jul 24 18:29:19.493554 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:19.493523 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" path="/var/lib/kubelet/pods/81fcdaba-b5bb-4265-9949-fa8a8d0a9191/volumes" Jul 24 18:29:25.853571 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:25.853536 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:29:28.146664 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:28.146626 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h"] Jul 24 18:29:28.152011 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:28.151986 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2542df-fwg2h"] Jul 24 18:29:29.493247 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:29.493214 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="428f039c-1908-42b0-8ada-36d25dc0c6f7" path="/var/lib/kubelet/pods/428f039c-1908-42b0-8ada-36d25dc0c6f7/volumes" Jul 24 18:29:35.258446 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258407 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74"] Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258735 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258746 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258753 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="428f039c-1908-42b0-8ada-36d25dc0c6f7" containerName="revoke-keys" Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258759 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="428f039c-1908-42b0-8ada-36d25dc0c6f7" containerName="revoke-keys" Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258798 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258803 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="79e18dfc-e25b-47b7-b493-a917cadd743c" containerName="istio-proxy" Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258811 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258817 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" Jul 24 18:29:35.258868 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258868 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="428f039c-1908-42b0-8ada-36d25dc0c6f7" containerName="revoke-keys" Jul 24 18:29:35.259161 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258876 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" Jul 24 18:29:35.259161 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.258883 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="81fcdaba-b5bb-4265-9949-fa8a8d0a9191" containerName="istio-proxy" Jul 24 18:29:35.261859 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.261843 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" Jul 24 18:29:35.264732 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.264713 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:29:35.264882 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.264806 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Jul 24 18:29:35.270164 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.270138 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74"] Jul 24 18:29:35.350536 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.350488 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df9b273-1ff4-4111-b21e-16584e6bb316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74\" (UID: \"1df9b273-1ff4-4111-b21e-16584e6bb316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" Jul 24 18:29:35.451571 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.451529 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df9b273-1ff4-4111-b21e-16584e6bb316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74\" (UID: \"1df9b273-1ff4-4111-b21e-16584e6bb316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" Jul 24 18:29:35.452210 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.452188 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df9b273-1ff4-4111-b21e-16584e6bb316-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74\" (UID: \"1df9b273-1ff4-4111-b21e-16584e6bb316\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" Jul 24 18:29:35.573135 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.573101 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" Jul 24 18:29:35.716225 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.716199 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74"] Jul 24 18:29:35.719057 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:29:35.719028 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1df9b273_1ff4_4111_b21e_16584e6bb316.slice/crio-1c8fa5407580c95f1daa6da73a38288c6ce08747cd4375d9394ec017a11e8a74 WatchSource:0}: Error finding container 1c8fa5407580c95f1daa6da73a38288c6ce08747cd4375d9394ec017a11e8a74: Status 404 returned error can't find the container with id 1c8fa5407580c95f1daa6da73a38288c6ce08747cd4375d9394ec017a11e8a74 Jul 24 18:29:35.764941 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:35.764911 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" event={"ID":"1df9b273-1ff4-4111-b21e-16584e6bb316","Type":"ContainerStarted","Data":"1c8fa5407580c95f1daa6da73a38288c6ce08747cd4375d9394ec017a11e8a74"} Jul 24 18:29:36.777591 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:36.777547 2559 generic.go:356] "Generic (PLEG): container finished" podID="1df9b273-1ff4-4111-b21e-16584e6bb316" containerID="6fdb179d8d5d1f1f292ca7d2018ef5d60919527260ed7e1758f425fad2c9a1f7" exitCode=0 Jul 24 18:29:36.778024 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:36.777634 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" event={"ID":"1df9b273-1ff4-4111-b21e-16584e6bb316","Type":"ContainerDied","Data":"6fdb179d8d5d1f1f292ca7d2018ef5d60919527260ed7e1758f425fad2c9a1f7"} Jul 24 18:29:37.916835 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:37.916778 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" Jul 24 18:29:38.075965 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:38.075930 2559 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df9b273-1ff4-4111-b21e-16584e6bb316-maas-api-service-ca\") pod \"1df9b273-1ff4-4111-b21e-16584e6bb316\" (UID: \"1df9b273-1ff4-4111-b21e-16584e6bb316\") " Jul 24 18:29:38.076297 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:38.076277 2559 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1df9b273-1ff4-4111-b21e-16584e6bb316-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1df9b273-1ff4-4111-b21e-16584e6bb316" (UID: "1df9b273-1ff4-4111-b21e-16584e6bb316"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 24 18:29:38.177393 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:38.177351 2559 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1df9b273-1ff4-4111-b21e-16584e6bb316-maas-api-service-ca\") on node \"ip-10-0-141-191.ec2.internal\" DevicePath \"\"" Jul 24 18:29:38.790194 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:38.790163 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" Jul 24 18:29:38.790383 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:38.790202 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74" event={"ID":"1df9b273-1ff4-4111-b21e-16584e6bb316","Type":"ContainerDied","Data":"1c8fa5407580c95f1daa6da73a38288c6ce08747cd4375d9394ec017a11e8a74"} Jul 24 18:29:38.790383 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:38.790250 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c8fa5407580c95f1daa6da73a38288c6ce08747cd4375d9394ec017a11e8a74" Jul 24 18:29:47.947766 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:47.947727 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-85d9db4d5d-kmslv"] Jul 24 18:29:50.335309 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:50.335276 2559 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74"] Jul 24 18:29:50.337427 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:50.337400 2559 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-f556c1-kzh74"] Jul 24 18:29:51.494284 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:29:51.494254 2559 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1df9b273-1ff4-4111-b21e-16584e6bb316" path="/var/lib/kubelet/pods/1df9b273-1ff4-4111-b21e-16584e6bb316/volumes" Jul 24 18:30:00.147293 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.147259 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l"] Jul 24 18:30:00.147751 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.147588 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df9b273-1ff4-4111-b21e-16584e6bb316" containerName="revoke-keys" Jul 24 18:30:00.147751 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.147599 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df9b273-1ff4-4111-b21e-16584e6bb316" containerName="revoke-keys" Jul 24 18:30:00.147751 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.147681 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="1df9b273-1ff4-4111-b21e-16584e6bb316" containerName="revoke-keys" Jul 24 18:30:00.152465 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.152444 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" Jul 24 18:30:00.162523 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.162500 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-dgh7n\"" Jul 24 18:30:00.162770 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.162755 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" Jul 24 18:30:00.169916 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.169876 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l"] Jul 24 18:30:00.319450 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.319418 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l"] Jul 24 18:30:00.322946 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:30:00.322919 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f05b64a_c42e_40e1_9922_17537021a329.slice/crio-d311ddbf39fee476c8e12cf0f564530319ad1ce3cc5889285477aaa458074b5b WatchSource:0}: Error finding container d311ddbf39fee476c8e12cf0f564530319ad1ce3cc5889285477aaa458074b5b: Status 404 returned error can't find the container with id d311ddbf39fee476c8e12cf0f564530319ad1ce3cc5889285477aaa458074b5b Jul 24 18:30:00.898527 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.898496 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" event={"ID":"3f05b64a-c42e-40e1-9922-17537021a329","Type":"ContainerStarted","Data":"707bcdf3bade56cee44405887972681d18eee5b25ec3301ec383886b5b47c2bc"} Jul 24 18:30:00.898711 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.898543 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" event={"ID":"3f05b64a-c42e-40e1-9922-17537021a329","Type":"ContainerStarted","Data":"d311ddbf39fee476c8e12cf0f564530319ad1ce3cc5889285477aaa458074b5b"} Jul 24 18:30:00.916980 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:00.916930 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" podStartSLOduration=0.916912817 podStartE2EDuration="916.912817ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:30:00.915968789 +0000 UTC m=+2825.955762829" watchObservedRunningTime="2026-07-24 18:30:00.916912817 +0000 UTC m=+2825.956706887" Jul 24 18:30:01.904477 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:01.904446 2559 generic.go:356] "Generic (PLEG): container finished" podID="3f05b64a-c42e-40e1-9922-17537021a329" containerID="707bcdf3bade56cee44405887972681d18eee5b25ec3301ec383886b5b47c2bc" exitCode=0 Jul 24 18:30:01.904969 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:01.904493 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" event={"ID":"3f05b64a-c42e-40e1-9922-17537021a329","Type":"ContainerDied","Data":"707bcdf3bade56cee44405887972681d18eee5b25ec3301ec383886b5b47c2bc"} Jul 24 18:30:02.920095 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:02.920070 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xdbrr_6c4537aa-c3c0-4d96-ab42-0a16dca59c48/manager/0.log" Jul 24 18:30:03.047309 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:03.047285 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pvvnn_46c58efc-9286-443c-abfe-e03a81664413/manager/0.log" Jul 24 18:30:03.065095 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:03.065072 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" Jul 24 18:30:03.538038 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:03.538012 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf4bd545-2tbcp_08178467-f23e-4195-b654-953476d882dd/copy-manifests/0.log" Jul 24 18:30:03.548845 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:03.548817 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf4bd545-2tbcp_08178467-f23e-4195-b654-953476d882dd/manager/0.log" Jul 24 18:30:03.916226 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:03.916202 2559 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" Jul 24 18:30:03.916410 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:03.916248 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748630-wr87l" event={"ID":"3f05b64a-c42e-40e1-9922-17537021a329","Type":"ContainerDied","Data":"d311ddbf39fee476c8e12cf0f564530319ad1ce3cc5889285477aaa458074b5b"} Jul 24 18:30:03.916410 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:03.916292 2559 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d311ddbf39fee476c8e12cf0f564530319ad1ce3cc5889285477aaa458074b5b" Jul 24 18:30:04.340126 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:04.340095 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748615-nf6qr_4ef608b7-af6d-4bfc-bdd4-8e2c38cac8d0/cleanup/0.log" Jul 24 18:30:04.445104 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:04.445078 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748630-wr87l_3f05b64a-c42e-40e1-9922-17537021a329/cleanup/0.log" Jul 24 18:30:05.877096 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:05.877070 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-kmslv_a98da47f-fd6b-4981-94b4-67d8ceb71d58/limitador/0.log" Jul 24 18:30:06.348478 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:06.348448 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k_58092870-06db-4d0f-ba90-12e6ecd13d95/istio-proxy/0.log" Jul 24 18:30:06.801286 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:06.801200 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tlnpt_d756e0ed-56b3-473c-8087-353d2eabd7b8/istio-proxy/1.log" Jul 24 18:30:07.138510 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:07.138476 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-758d7d5dd5-wq9dp_ed3b05b8-f53f-4148-8b50-dac538f09ad6/router/0.log" Jul 24 18:30:07.802493 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:07.802461 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f64dgw_cddf6ddc-17d2-461a-8663-e6616fb3c79b/main/0.log" Jul 24 18:30:13.008049 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.008009 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jqhx8/must-gather-rrc7z"] Jul 24 18:30:13.008453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.008332 2559 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f05b64a-c42e-40e1-9922-17537021a329" containerName="cleanup" Jul 24 18:30:13.008453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.008343 2559 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f05b64a-c42e-40e1-9922-17537021a329" containerName="cleanup" Jul 24 18:30:13.008453 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.008418 2559 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f05b64a-c42e-40e1-9922-17537021a329" containerName="cleanup" Jul 24 18:30:13.011644 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.011628 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqhx8/must-gather-rrc7z" Jul 24 18:30:13.014528 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.014507 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jqhx8\"/\"openshift-service-ca.crt\"" Jul 24 18:30:13.015684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.015666 2559 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jqhx8\"/\"default-dockercfg-sqr6w\"" Jul 24 18:30:13.015812 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.015672 2559 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jqhx8\"/\"kube-root-ca.crt\"" Jul 24 18:30:13.022663 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.022642 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqhx8/must-gather-rrc7z"] Jul 24 18:30:13.078125 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.078088 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73b6795d-32c8-4d7a-9515-bf217586e267-must-gather-output\") pod \"must-gather-rrc7z\" (UID: \"73b6795d-32c8-4d7a-9515-bf217586e267\") " pod="openshift-must-gather-jqhx8/must-gather-rrc7z" Jul 24 18:30:13.078337 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.078235 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htrvn\" (UniqueName: \"kubernetes.io/projected/73b6795d-32c8-4d7a-9515-bf217586e267-kube-api-access-htrvn\") pod \"must-gather-rrc7z\" (UID: \"73b6795d-32c8-4d7a-9515-bf217586e267\") " pod="openshift-must-gather-jqhx8/must-gather-rrc7z" Jul 24 18:30:13.179538 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.179495 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htrvn\" (UniqueName: \"kubernetes.io/projected/73b6795d-32c8-4d7a-9515-bf217586e267-kube-api-access-htrvn\") pod \"must-gather-rrc7z\" (UID: \"73b6795d-32c8-4d7a-9515-bf217586e267\") " pod="openshift-must-gather-jqhx8/must-gather-rrc7z" Jul 24 18:30:13.179538 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.179538 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73b6795d-32c8-4d7a-9515-bf217586e267-must-gather-output\") pod \"must-gather-rrc7z\" (UID: \"73b6795d-32c8-4d7a-9515-bf217586e267\") " pod="openshift-must-gather-jqhx8/must-gather-rrc7z" Jul 24 18:30:13.179910 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.179894 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/73b6795d-32c8-4d7a-9515-bf217586e267-must-gather-output\") pod \"must-gather-rrc7z\" (UID: \"73b6795d-32c8-4d7a-9515-bf217586e267\") " pod="openshift-must-gather-jqhx8/must-gather-rrc7z" Jul 24 18:30:13.188659 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.188625 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-htrvn\" (UniqueName: \"kubernetes.io/projected/73b6795d-32c8-4d7a-9515-bf217586e267-kube-api-access-htrvn\") pod \"must-gather-rrc7z\" (UID: \"73b6795d-32c8-4d7a-9515-bf217586e267\") " pod="openshift-must-gather-jqhx8/must-gather-rrc7z" Jul 24 18:30:13.320719 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.320685 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqhx8/must-gather-rrc7z" Jul 24 18:30:13.463445 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.463412 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqhx8/must-gather-rrc7z"] Jul 24 18:30:13.465973 ip-10-0-141-191 kubenswrapper[2559]: W0724 18:30:13.465940 2559 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73b6795d_32c8_4d7a_9515_bf217586e267.slice/crio-70330b55dfa14d0d845fbc98477d5a6aa909940c0107f65cb4dff57687ff8eb3 WatchSource:0}: Error finding container 70330b55dfa14d0d845fbc98477d5a6aa909940c0107f65cb4dff57687ff8eb3: Status 404 returned error can't find the container with id 70330b55dfa14d0d845fbc98477d5a6aa909940c0107f65cb4dff57687ff8eb3 Jul 24 18:30:13.967582 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:13.967544 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqhx8/must-gather-rrc7z" event={"ID":"73b6795d-32c8-4d7a-9515-bf217586e267","Type":"ContainerStarted","Data":"70330b55dfa14d0d845fbc98477d5a6aa909940c0107f65cb4dff57687ff8eb3"} Jul 24 18:30:14.976617 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:14.976581 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqhx8/must-gather-rrc7z" event={"ID":"73b6795d-32c8-4d7a-9515-bf217586e267","Type":"ContainerStarted","Data":"c7f041b25413689f310edb16b352cb1f08f216ed4bb08d499a0aebb3cc84e0e6"} Jul 24 18:30:14.977063 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:14.976631 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqhx8/must-gather-rrc7z" event={"ID":"73b6795d-32c8-4d7a-9515-bf217586e267","Type":"ContainerStarted","Data":"77d3bbdcd0736b39309dce7fb2495dd8f32be9e282e5fc0a0234325dacc96c56"} Jul 24 18:30:14.996767 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:14.996716 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jqhx8/must-gather-rrc7z" podStartSLOduration=2.265773235 podStartE2EDuration="2.996698541s" totalImagesPullingTime="730.925306ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 18:30:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:30:14.993809378 +0000 UTC m=+2840.033603411" watchObservedRunningTime="2026-07-24 18:30:14.996698541 +0000 UTC m=+2840.036492581" Jul 24 18:30:15.864433 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:15.864401 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zh5f7_b88c2b38-14f5-4ef4-8e28-76663ce9e34f/global-pull-secret-syncer/0.log" Jul 24 18:30:16.006953 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:16.006923 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zkd24_c63c5161-5c4c-44e8-991b-603b5c3b24b9/konnectivity-agent/0.log" Jul 24 18:30:16.098212 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:16.098180 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-191.ec2.internal_d2a76f45062ee97c58334419b354384f/haproxy/0.log" Jul 24 18:30:21.516416 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:21.516379 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-85d9db4d5d-kmslv_a98da47f-fd6b-4981-94b4-67d8ceb71d58/limitador/0.log" Jul 24 18:30:23.486878 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:23.486257 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-cvpqk_02e6ee41-b12e-42ee-9589-f0435e5ae956/cluster-monitoring-operator/0.log" Jul 24 18:30:23.893729 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:23.893654 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slxvw_20622eb3-3ce9-498a-b95a-fdc5c59b7e71/node-exporter/0.log" Jul 24 18:30:23.918660 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:23.918550 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slxvw_20622eb3-3ce9-498a-b95a-fdc5c59b7e71/kube-rbac-proxy/0.log" Jul 24 18:30:23.951627 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:23.951594 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-slxvw_20622eb3-3ce9-498a-b95a-fdc5c59b7e71/init-textfile/0.log" Jul 24 18:30:25.908599 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:25.908556 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-74c9f884b7-26s9g_edfb93c0-838e-40ee-8fda-324733002b82/networking-console-plugin/0.log" Jul 24 18:30:26.134956 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.134917 2559 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m"] Jul 24 18:30:26.138611 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.138582 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.147421 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.147375 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m"] Jul 24 18:30:26.219968 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.219875 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-sys\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.220254 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.220230 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-podres\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.220414 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.220398 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-proc\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.220561 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.220545 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-lib-modules\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.220698 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.220683 2559 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w852r\" (UniqueName: \"kubernetes.io/projected/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-kube-api-access-w852r\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.321640 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.321598 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-sys\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.321857 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.321680 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-podres\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.321857 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.321724 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-proc\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.321857 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.321743 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-sys\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.321857 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.321774 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-lib-modules\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.321857 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.321849 2559 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w852r\" (UniqueName: \"kubernetes.io/projected/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-kube-api-access-w852r\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.322114 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.321860 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-proc\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.322114 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.321927 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-podres\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.322114 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.322029 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-lib-modules\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.331594 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.331569 2559 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w852r\" (UniqueName: \"kubernetes.io/projected/a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f-kube-api-access-w852r\") pod \"perf-node-gather-daemonset-dqz9m\" (UID: \"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f\") " pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.449021 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.448976 2559 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:26.573255 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.573174 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/1.log" Jul 24 18:30:26.580682 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.580620 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-dz2pl_c64451b6-8365-4a7a-8512-baaee772473d/console-operator/2.log" Jul 24 18:30:26.636443 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:26.636408 2559 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m"] Jul 24 18:30:27.071711 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:27.071670 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" event={"ID":"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f","Type":"ContainerStarted","Data":"bf4ed909506b5a9321811fcf033cf425cb266804ea0a7fd4303168ef712548ac"} Jul 24 18:30:27.072221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:27.071753 2559 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" event={"ID":"a2d9f5b7-48a9-40a0-bac3-ee2d1c51f04f","Type":"ContainerStarted","Data":"2b7166ed5264a36ba5645b48909c9ece29be88611b0c7a57658bcd37686460f7"} Jul 24 18:30:27.072221 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:27.071825 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:27.783255 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:27.783222 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-wqnfg_72416b1c-f944-4c29-b8fe-5a3b053825f7/volume-data-source-validator/0.log" Jul 24 18:30:28.880281 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:28.880257 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cbm78_d642fa46-5c47-4435-955b-1f73afb4fb81/dns/0.log" Jul 24 18:30:28.902321 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:28.902285 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-cbm78_d642fa46-5c47-4435-955b-1f73afb4fb81/kube-rbac-proxy/0.log" Jul 24 18:30:28.974289 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:28.974264 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gnjpf_3076bbf1-a934-4c5d-a679-a3aa1c824daf/dns-node-resolver/0.log" Jul 24 18:30:29.610121 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:29.610084 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fsf7z_b00377e3-0fe2-4fe7-89cb-c892e233959f/node-ca/0.log" Jul 24 18:30:30.594944 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:30.594913 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k_58092870-06db-4d0f-ba90-12e6ecd13d95/istio-proxy/0.log" Jul 24 18:30:30.853151 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:30.853076 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tlnpt_d756e0ed-56b3-473c-8087-353d2eabd7b8/istio-proxy/0.log" Jul 24 18:30:30.861936 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:30.861908 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tlnpt_d756e0ed-56b3-473c-8087-353d2eabd7b8/istio-proxy/1.log" Jul 24 18:30:30.952537 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:30.952509 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-758d7d5dd5-wq9dp_ed3b05b8-f53f-4148-8b50-dac538f09ad6/router/0.log" Jul 24 18:30:31.822127 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:31.822095 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hnvlj_f5e968a2-537a-4759-bc98-e669d89fe627/serve-healthcheck-canary/0.log" Jul 24 18:30:32.344746 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:32.344710 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-k2526_b9c8447a-8f08-4d76-a7d3-103f60b7e495/insights-operator/0.log" Jul 24 18:30:32.345349 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:32.345331 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5885fdb759-k2526_b9c8447a-8f08-4d76-a7d3-103f60b7e495/insights-operator/1.log" Jul 24 18:30:32.367357 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:32.367325 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24c26_d6bce77f-3e2f-40af-8109-b52446e53c4a/kube-rbac-proxy/0.log" Jul 24 18:30:32.391323 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:32.391296 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24c26_d6bce77f-3e2f-40af-8109-b52446e53c4a/exporter/0.log" Jul 24 18:30:32.417330 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:32.417300 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-24c26_d6bce77f-3e2f-40af-8109-b52446e53c4a/extractor/0.log" Jul 24 18:30:33.096610 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:33.096576 2559 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" Jul 24 18:30:33.117095 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:33.117030 2559 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jqhx8/perf-node-gather-daemonset-dqz9m" podStartSLOduration=7.11700935 podStartE2EDuration="7.11700935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 18:30:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 18:30:27.090881087 +0000 UTC m=+2852.130675129" watchObservedRunningTime="2026-07-24 18:30:33.11700935 +0000 UTC m=+2858.156803392" Jul 24 18:30:34.952175 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:34.952149 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-xdbrr_6c4537aa-c3c0-4d96-ab42-0a16dca59c48/manager/0.log" Jul 24 18:30:35.222770 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:35.222742 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-pvvnn_46c58efc-9286-443c-abfe-e03a81664413/manager/0.log" Jul 24 18:30:35.495116 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:35.495033 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf4bd545-2tbcp_08178467-f23e-4195-b654-953476d882dd/manager/0.log" Jul 24 18:30:35.522660 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:35.522632 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbf4bd545-2tbcp_08178467-f23e-4195-b654-953476d882dd/copy-manifests/0.log" Jul 24 18:30:36.599448 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:36.599413 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k_58092870-06db-4d0f-ba90-12e6ecd13d95/istio-proxy/0.log" Jul 24 18:30:36.807494 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:36.807397 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tlnpt_d756e0ed-56b3-473c-8087-353d2eabd7b8/istio-proxy/0.log" Jul 24 18:30:36.815684 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:36.815646 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tlnpt_d756e0ed-56b3-473c-8087-353d2eabd7b8/istio-proxy/1.log" Jul 24 18:30:36.869046 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:36.869022 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-758d7d5dd5-wq9dp_ed3b05b8-f53f-4148-8b50-dac538f09ad6/router/0.log" Jul 24 18:30:36.981561 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:36.981533 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-9f897544f-crvdf_bd9785f3-bf18-4643-8a7f-8961b0aecd58/manager/0.log" Jul 24 18:30:38.354366 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:38.354340 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-47v2k_58092870-06db-4d0f-ba90-12e6ecd13d95/istio-proxy/0.log" Jul 24 18:30:38.577656 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:38.577623 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tlnpt_d756e0ed-56b3-473c-8087-353d2eabd7b8/istio-proxy/0.log" Jul 24 18:30:38.587425 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:38.587379 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-tlnpt_d756e0ed-56b3-473c-8087-353d2eabd7b8/istio-proxy/1.log" Jul 24 18:30:38.655377 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:38.655348 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-758d7d5dd5-wq9dp_ed3b05b8-f53f-4148-8b50-dac538f09ad6/router/0.log" Jul 24 18:30:42.431910 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:42.431872 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf_6f5e6ac9-6056-4018-a77e-33ac5e69cf72/kube-storage-version-migrator-operator/1.log" Jul 24 18:30:42.433030 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:42.433012 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-kqhvf_6f5e6ac9-6056-4018-a77e-33ac5e69cf72/kube-storage-version-migrator-operator/0.log" Jul 24 18:30:43.424839 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:43.424774 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-29ff9_8bcdff46-acf9-4858-aee1-d92f3219d7e9/kube-multus/0.log" Jul 24 18:30:43.456247 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:43.456219 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96j5z_0cab09e2-baba-4a01-a58f-53c91600107f/kube-multus-additional-cni-plugins/0.log" Jul 24 18:30:43.491503 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:43.491470 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96j5z_0cab09e2-baba-4a01-a58f-53c91600107f/egress-router-binary-copy/0.log" Jul 24 18:30:43.523614 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:43.523580 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96j5z_0cab09e2-baba-4a01-a58f-53c91600107f/cni-plugins/0.log" Jul 24 18:30:43.553962 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:43.553933 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96j5z_0cab09e2-baba-4a01-a58f-53c91600107f/bond-cni-plugin/0.log" Jul 24 18:30:43.579073 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:43.579050 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96j5z_0cab09e2-baba-4a01-a58f-53c91600107f/routeoverride-cni/0.log" Jul 24 18:30:43.607711 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:43.607662 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96j5z_0cab09e2-baba-4a01-a58f-53c91600107f/whereabouts-cni-bincopy/0.log" Jul 24 18:30:43.637141 ip-10-0-141-191 kubenswrapper[2559]: I0724 18:30:43.637108 2559 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-96j5z_0cab09e2-baba-4a01-a58f-53c91600107f/whereabouts-cni/0.log"