Jul 30 14:46:16.045180 ip-10-0-130-38 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 30 14:46:16.045195 ip-10-0-130-38 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 30 14:46:16.045222 ip-10-0-130-38 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 30 14:46:16.045573 ip-10-0-130-38 systemd[1]: Failed to start Kubernetes Kubelet. Jul 30 14:46:25.272195 ip-10-0-130-38 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Jul 30 14:46:25.272210 ip-10-0-130-38 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot fd2c33f8195149c4af37a74227bc0556 -- Jul 30 14:47:52.901448 ip-10-0-130-38 systemd[1]: Starting Kubernetes Kubelet... Jul 30 14:47:53.327001 ip-10-0-130-38 kubenswrapper[2564]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 14:47:53.327001 ip-10-0-130-38 kubenswrapper[2564]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 30 14:47:53.327001 ip-10-0-130-38 kubenswrapper[2564]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 14:47:53.327001 ip-10-0-130-38 kubenswrapper[2564]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 30 14:47:53.327001 ip-10-0-130-38 kubenswrapper[2564]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 30 14:47:53.328916 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.328824 2564 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 30 14:47:53.334225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334205 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 14:47:53.334225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334224 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334229 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334234 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334237 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334240 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334243 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334247 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334250 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334252 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334255 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334263 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334266 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334269 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334271 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334274 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334276 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334279 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334282 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334284 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 14:47:53.334300 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334287 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334289 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334292 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334296 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334300 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334303 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334306 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334309 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334311 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334314 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334317 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334319 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334322 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334324 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334327 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334329 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334332 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334334 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334337 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 14:47:53.334748 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334339 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334342 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334345 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334347 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334350 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334352 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334355 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334357 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334360 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334362 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334365 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334370 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334373 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334376 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334379 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334384 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334387 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334390 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334392 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334395 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 14:47:53.335220 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334398 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334401 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334403 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334406 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334408 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334411 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334413 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334416 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334418 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334421 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334423 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334426 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334429 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334431 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334434 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334436 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334439 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334441 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334444 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334446 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 14:47:53.335711 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334449 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334452 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334455 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334458 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334460 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334462 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334465 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334468 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334471 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334474 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334477 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334480 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334482 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334485 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334488 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334490 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334492 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334495 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334498 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.334500 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 14:47:53.336207 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335272 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335280 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335283 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335286 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335289 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335292 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335295 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335298 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335301 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335303 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335306 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335308 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335311 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335314 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335316 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335319 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335321 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335324 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335326 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335329 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 14:47:53.336701 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335332 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335335 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335337 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335340 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335343 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335345 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335348 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335351 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335353 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335357 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335360 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335363 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335367 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335370 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335373 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335376 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335378 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335381 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335384 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 14:47:53.337286 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335387 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335389 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335392 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335394 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335396 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335399 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335402 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335404 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335407 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335409 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335412 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335414 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335418 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335421 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335423 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335426 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335428 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335431 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335433 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335436 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 14:47:53.337754 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335439 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335442 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335444 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335446 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335449 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335451 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335454 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335456 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335458 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335461 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335463 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335466 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335469 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335472 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335477 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335480 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335484 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335487 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335490 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 14:47:53.338283 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335493 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335496 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335499 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335501 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335504 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335508 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335511 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335513 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335516 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335518 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335521 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335523 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335526 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335528 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335531 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335533 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335536 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335539 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335542 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335544 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 14:47:53.338747 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335547 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335938 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335943 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335946 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335949 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335952 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335955 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335959 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335962 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335965 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335968 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335970 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335973 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335975 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335978 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335981 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335985 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335990 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335993 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.335996 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 14:47:53.339249 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336000 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336002 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336005 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336008 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336010 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336013 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336016 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336018 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336021 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336023 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336026 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336028 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336031 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336033 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336036 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336038 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336041 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336044 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336048 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336053 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 14:47:53.339761 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336055 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336058 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336060 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336063 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336065 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336068 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336084 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336087 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336089 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336093 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336096 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336098 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336101 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336103 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336106 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336108 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336111 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336113 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336116 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336119 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 14:47:53.340266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336121 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336124 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336127 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336130 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336133 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336135 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336138 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336141 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336144 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336146 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336149 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336154 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336157 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336159 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336162 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336165 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336167 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336170 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336172 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 14:47:53.340760 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336176 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336179 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336184 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336187 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336190 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336193 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336195 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336198 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336200 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336203 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336206 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336208 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336211 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336214 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336216 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336219 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336221 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336224 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336227 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 14:47:53.341272 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336229 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336232 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336313 2564 flags.go:64] FLAG: --address="0.0.0.0" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336321 2564 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336341 2564 flags.go:64] FLAG: --anonymous-auth="true" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336348 2564 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336352 2564 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336356 2564 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336360 2564 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336365 2564 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336368 2564 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336371 2564 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336374 2564 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336378 2564 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336381 2564 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336384 2564 flags.go:64] FLAG: --cgroup-root="" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336387 2564 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336390 2564 flags.go:64] FLAG: --client-ca-file="" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336394 2564 flags.go:64] FLAG: --cloud-provider="external" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336397 2564 flags.go:64] FLAG: --cluster-dns="[]" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336401 2564 flags.go:64] FLAG: --cluster-domain="" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336404 2564 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336407 2564 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336410 2564 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 30 14:47:53.341733 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336414 2564 flags.go:64] FLAG: --container-log-max-files="5" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336418 2564 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336421 2564 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336424 2564 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336427 2564 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336430 2564 flags.go:64] FLAG: --contention-profiling="false" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336433 2564 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336436 2564 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336439 2564 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336442 2564 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336446 2564 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336449 2564 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336452 2564 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336456 2564 flags.go:64] FLAG: --enable-load-reader="false" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336460 2564 flags.go:64] FLAG: --enable-server="true" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336471 2564 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336476 2564 flags.go:64] FLAG: --event-burst="100" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336479 2564 flags.go:64] FLAG: --event-qps="50" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336482 2564 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336485 2564 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336488 2564 flags.go:64] FLAG: --eviction-hard="" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336492 2564 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336495 2564 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336499 2564 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336502 2564 flags.go:64] FLAG: --eviction-soft="" Jul 30 14:47:53.342316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336505 2564 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336507 2564 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336510 2564 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336513 2564 flags.go:64] FLAG: --experimental-mounter-path="" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336516 2564 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336519 2564 flags.go:64] FLAG: --fail-swap-on="true" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336522 2564 flags.go:64] FLAG: --feature-gates="" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336526 2564 flags.go:64] FLAG: --file-check-frequency="20s" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336529 2564 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336533 2564 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336536 2564 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336539 2564 flags.go:64] FLAG: --healthz-port="10248" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336543 2564 flags.go:64] FLAG: --help="false" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336545 2564 flags.go:64] FLAG: --hostname-override="ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336549 2564 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336552 2564 flags.go:64] FLAG: --http-check-frequency="20s" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336555 2564 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336558 2564 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336561 2564 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336564 2564 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336570 2564 flags.go:64] FLAG: --image-service-endpoint="" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336573 2564 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336576 2564 flags.go:64] FLAG: --kube-api-burst="100" Jul 30 14:47:53.342926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336579 2564 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336582 2564 flags.go:64] FLAG: --kube-api-qps="50" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336585 2564 flags.go:64] FLAG: --kube-reserved="" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336588 2564 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336591 2564 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336594 2564 flags.go:64] FLAG: --kubelet-cgroups="" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336597 2564 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336601 2564 flags.go:64] FLAG: --lock-file="" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336604 2564 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336607 2564 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336610 2564 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336616 2564 flags.go:64] FLAG: --log-json-split-stream="false" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336619 2564 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336621 2564 flags.go:64] FLAG: --log-text-split-stream="false" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336624 2564 flags.go:64] FLAG: --logging-format="text" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336628 2564 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336631 2564 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336634 2564 flags.go:64] FLAG: --manifest-url="" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336637 2564 flags.go:64] FLAG: --manifest-url-header="" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336642 2564 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336645 2564 flags.go:64] FLAG: --max-open-files="1000000" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336649 2564 flags.go:64] FLAG: --max-pods="110" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336652 2564 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336655 2564 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336658 2564 flags.go:64] FLAG: --memory-manager-policy="None" Jul 30 14:47:53.343699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336661 2564 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336664 2564 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336666 2564 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336669 2564 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336678 2564 flags.go:64] FLAG: --node-status-max-images="50" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336681 2564 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336684 2564 flags.go:64] FLAG: --oom-score-adj="-999" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336687 2564 flags.go:64] FLAG: --pod-cidr="" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336698 2564 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336703 2564 flags.go:64] FLAG: --pod-manifest-path="" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336707 2564 flags.go:64] FLAG: --pod-max-pids="-1" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336710 2564 flags.go:64] FLAG: --pods-per-core="0" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336713 2564 flags.go:64] FLAG: --port="10250" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336716 2564 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336719 2564 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08b09d9d3b8e9cdb4" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336722 2564 flags.go:64] FLAG: --qos-reserved="" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336725 2564 flags.go:64] FLAG: --read-only-port="10255" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336728 2564 flags.go:64] FLAG: --register-node="true" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336731 2564 flags.go:64] FLAG: --register-with-taints="" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336739 2564 flags.go:64] FLAG: --registry-burst="10" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336742 2564 flags.go:64] FLAG: --registry-qps="5" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336744 2564 flags.go:64] FLAG: --reserved-cpus="" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336747 2564 flags.go:64] FLAG: --reserved-memory="" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336751 2564 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336755 2564 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 30 14:47:53.344810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336758 2564 flags.go:64] FLAG: --rotate-certificates="false" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336761 2564 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336764 2564 flags.go:64] FLAG: --runonce="false" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336767 2564 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336770 2564 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336773 2564 flags.go:64] FLAG: --seccomp-default="false" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336777 2564 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336780 2564 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336783 2564 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336786 2564 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336789 2564 flags.go:64] FLAG: --storage-driver-password="root" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336792 2564 flags.go:64] FLAG: --storage-driver-secure="false" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336796 2564 flags.go:64] FLAG: --storage-driver-table="stats" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336799 2564 flags.go:64] FLAG: --storage-driver-user="root" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336802 2564 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336806 2564 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336809 2564 flags.go:64] FLAG: --system-cgroups="" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336812 2564 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336817 2564 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336820 2564 flags.go:64] FLAG: --tls-cert-file="" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336825 2564 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336829 2564 flags.go:64] FLAG: --tls-min-version="" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336832 2564 flags.go:64] FLAG: --tls-private-key-file="" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336834 2564 flags.go:64] FLAG: --topology-manager-policy="none" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336837 2564 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 30 14:47:53.345953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336840 2564 flags.go:64] FLAG: --topology-manager-scope="container" Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336843 2564 flags.go:64] FLAG: --v="2" Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336848 2564 flags.go:64] FLAG: --version="false" Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336852 2564 flags.go:64] FLAG: --vmodule="" Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336856 2564 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.336859 2564 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336989 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336993 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336996 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.336999 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337002 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337005 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337008 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337011 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337013 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337016 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337019 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337021 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337024 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337027 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337030 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 14:47:53.346742 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337032 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337035 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337038 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337041 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337044 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337047 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337051 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337054 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337057 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337061 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337063 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337066 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337069 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337083 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337086 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337088 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337092 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337094 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337098 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 14:47:53.347560 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337102 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337105 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337108 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337111 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337114 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337116 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337119 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337122 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337124 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337127 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337130 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337133 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337136 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337138 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337141 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337144 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337146 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337149 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337152 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337156 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 14:47:53.348408 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337158 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337161 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337163 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337166 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337168 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337171 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337173 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337176 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337178 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337180 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337183 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337185 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337189 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337193 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337196 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337199 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337201 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337204 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337207 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337210 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 14:47:53.349225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337212 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337215 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337217 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337220 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337223 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337225 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337228 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337231 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337233 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337236 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337239 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337243 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337245 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337248 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337250 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337253 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337255 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337258 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337260 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337263 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 14:47:53.349879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337265 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337268 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337270 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337273 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.337275 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.337915 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.346262 2564 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.346285 2564 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.347190 2564 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.347205 2564 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347327 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347337 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347343 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347347 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347352 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 14:47:53.350483 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347358 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347363 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347368 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347373 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347377 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347382 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347386 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347392 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347396 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347401 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347406 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347410 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347414 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347419 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347423 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347427 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347432 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347437 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347441 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347446 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 14:47:53.350858 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347450 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347454 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347458 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347462 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347469 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347473 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347478 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347483 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347488 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347492 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347496 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347501 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347505 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347509 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347513 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347517 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347522 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347526 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347530 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347534 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 14:47:53.351361 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347539 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347543 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347547 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347552 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347556 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347560 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347564 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347568 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347572 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347576 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347581 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347585 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347589 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347596 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347603 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347608 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347613 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347618 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347622 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347627 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 14:47:53.351850 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347631 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347635 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347640 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347646 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347652 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347658 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347662 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347669 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347674 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347679 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347684 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347689 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347693 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347698 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347703 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347707 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347712 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347716 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347720 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 14:47:53.352359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347724 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347729 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347733 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347738 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347743 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347747 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347752 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347756 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347760 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347765 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347770 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347775 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347780 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347785 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.347789 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 14:47:53.352853 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.347797 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348025 2564 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348034 2564 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348038 2564 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348042 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348047 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348051 2564 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348063 2564 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348068 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348087 2564 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348092 2564 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348096 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348100 2564 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348104 2564 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348108 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348112 2564 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348117 2564 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348121 2564 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348126 2564 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 30 14:47:53.353266 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348130 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348134 2564 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348139 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348143 2564 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348147 2564 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348151 2564 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348155 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348159 2564 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348164 2564 feature_gate.go:328] unrecognized feature gate: Example2 Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348168 2564 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348183 2564 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348188 2564 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348192 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348196 2564 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348201 2564 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348205 2564 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348209 2564 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348213 2564 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348218 2564 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348222 2564 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 30 14:47:53.353732 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348226 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348232 2564 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348236 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348240 2564 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348244 2564 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348249 2564 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348253 2564 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348257 2564 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348261 2564 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348266 2564 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348270 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348274 2564 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348279 2564 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348283 2564 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348287 2564 feature_gate.go:328] unrecognized feature gate: Example Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348292 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348296 2564 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348302 2564 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348306 2564 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348310 2564 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 30 14:47:53.354237 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348314 2564 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348318 2564 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348325 2564 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348338 2564 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348344 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348348 2564 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348352 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348356 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348361 2564 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348365 2564 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348369 2564 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348373 2564 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348380 2564 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348386 2564 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348391 2564 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348396 2564 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348402 2564 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348407 2564 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 30 14:47:53.354720 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348411 2564 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348415 2564 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348419 2564 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348423 2564 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348427 2564 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348431 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348435 2564 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348440 2564 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348444 2564 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348448 2564 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348453 2564 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348457 2564 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348461 2564 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348474 2564 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348478 2564 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348482 2564 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348486 2564 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348490 2564 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348503 2564 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348508 2564 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 30 14:47:53.355214 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348512 2564 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 30 14:47:53.355695 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348517 2564 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 30 14:47:53.355695 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:53.348521 2564 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 30 14:47:53.355695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.348530 2564 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 30 14:47:53.355695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.349096 2564 server.go:962] "Client rotation is on, will bootstrap in background" Jul 30 14:47:53.355695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.352648 2564 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 30 14:47:53.355695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.354181 2564 server.go:1019] "Starting client certificate rotation" Jul 30 14:47:53.355695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.354281 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 14:47:53.355695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.354317 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 30 14:47:53.357140 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.357125 2564 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 14:47:53.360672 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.360656 2564 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 30 14:47:53.371896 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.371863 2564 log.go:25] "Validated CRI v1 runtime API" Jul 30 14:47:53.373961 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.373947 2564 log.go:25] "Validated CRI v1 image API" Jul 30 14:47:53.375391 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.375376 2564 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 30 14:47:53.380936 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.380913 2564 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 8ce51a70-6b89-4de9-8e13-2f6599c6a982:/dev/nvme0n1p4 b5c5aba3-a1de-4ec6-b55a-4085665ce8f2:/dev/nvme0n1p3] Jul 30 14:47:53.381026 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.380936 2564 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 30 14:47:53.383610 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.383589 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 14:47:53.386619 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.386504 2564 manager.go:216] Machine: {Timestamp:2026-07-30 14:47:53.38532947 +0000 UTC m=+0.368612798 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090609 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec25b4de2e01512e98289dfeb70a7615 SystemUUID:ec25b4de-2e01-512e-9828-9dfeb70a7615 BootID:fd2c33f8-1951-49c4-af37-a74227bc0556 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:98:3d:8c:b3:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:98:3d:8c:b3:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:06:86:2e:9e:ea:98 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 30 14:47:53.386619 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.386614 2564 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 30 14:47:53.386741 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.386700 2564 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 30 14:47:53.389605 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.389579 2564 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 30 14:47:53.389749 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.389608 2564 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-38.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 30 14:47:53.390388 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.390376 2564 topology_manager.go:138] "Creating topology manager with none policy" Jul 30 14:47:53.390430 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.390389 2564 container_manager_linux.go:309] "Creating device plugin manager" Jul 30 14:47:53.390430 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.390397 2564 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 14:47:53.391085 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.391057 2564 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 30 14:47:53.391939 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.391928 2564 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 30 14:47:53.393767 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.393755 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 30 14:47:53.393912 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.393902 2564 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 30 14:47:53.393952 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.393934 2564 kubelet.go:486] "Attempting to sync node with API server" Jul 30 14:47:53.393952 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.393951 2564 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 30 14:47:53.394027 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.393968 2564 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 30 14:47:53.394027 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.393977 2564 kubelet.go:398] "Adding apiserver pod source" Jul 30 14:47:53.394027 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.393985 2564 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 30 14:47:53.394595 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.394581 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 14:47:53.394660 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.394601 2564 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 30 14:47:53.399336 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.399308 2564 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 30 14:47:53.401952 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.401930 2564 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 30 14:47:53.402028 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.401968 2564 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 30 14:47:53.403660 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403645 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 30 14:47:53.403660 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403662 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 30 14:47:53.403660 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403668 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 30 14:47:53.403660 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403673 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 30 14:47:53.403868 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403682 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 30 14:47:53.403868 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403688 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 30 14:47:53.403868 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403696 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 30 14:47:53.403868 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403701 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 30 14:47:53.403868 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403708 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 30 14:47:53.403868 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403714 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 30 14:47:53.403868 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403728 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 30 14:47:53.403868 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.403734 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 30 14:47:53.405267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.405250 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 30 14:47:53.405267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.405263 2564 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 30 14:47:53.408392 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.408378 2564 server.go:1268] "Started kubelet" Jul 30 14:47:53.408527 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.408483 2564 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 30 14:47:53.409025 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.408961 2564 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 30 14:47:53.409166 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.409051 2564 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 30 14:47:53.409395 ip-10-0-130-38 systemd[1]: Started Kubernetes Kubelet. Jul 30 14:47:53.409728 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.409701 2564 server.go:310] "Adding debug handlers to kubelet server" Jul 30 14:47:53.410617 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.410597 2564 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 30 14:47:53.411617 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.411596 2564 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-38.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 30 14:47:53.411617 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.411596 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 30 14:47:53.411617 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.411598 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 30 14:47:53.413923 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.413904 2564 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 30 14:47:53.416953 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.416933 2564 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 30 14:47:53.417045 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.416965 2564 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 30 14:47:53.417045 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.417027 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:53.417763 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.414818 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-38.ec2.internal.18c718a4cf5ca285 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-38.ec2.internal,UID:ip-10-0-130-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-38.ec2.internal,},FirstTimestamp:2026-07-30 14:47:53.408348805 +0000 UTC m=+0.391632132,LastTimestamp:2026-07-30 14:47:53.408348805 +0000 UTC m=+0.391632132,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-38.ec2.internal,}" Jul 30 14:47:53.417857 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.417804 2564 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 30 14:47:53.417857 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.417822 2564 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 30 14:47:53.417857 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.417847 2564 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 30 14:47:53.418000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.417977 2564 reconstruct.go:97] "Volume reconstruction finished" Jul 30 14:47:53.418000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.417985 2564 reconciler.go:29] "Reconciler: start to sync state" Jul 30 14:47:53.418346 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.418329 2564 factory.go:153] Registering CRI-O factory Jul 30 14:47:53.418417 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.418353 2564 factory.go:223] Registration of the crio container factory successfully Jul 30 14:47:53.418417 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.418402 2564 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 30 14:47:53.418417 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.418411 2564 factory.go:55] Registering systemd factory Jul 30 14:47:53.418417 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.418419 2564 factory.go:223] Registration of the systemd container factory successfully Jul 30 14:47:53.418592 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.418449 2564 factory.go:103] Registering Raw factory Jul 30 14:47:53.418592 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.418466 2564 manager.go:1196] Started watching for new ooms in manager Jul 30 14:47:53.418990 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.418968 2564 manager.go:349] Starting recovery of all containers Jul 30 14:47:53.420824 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.420779 2564 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-38.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 30 14:47:53.420824 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.420804 2564 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 30 14:47:53.425139 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.425109 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 30 14:47:53.430134 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.430103 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6gbn" Jul 30 14:47:53.431580 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.431556 2564 manager.go:354] Recovery completed Jul 30 14:47:53.437008 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.436984 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w6gbn" Jul 30 14:47:53.437771 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.437759 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 14:47:53.439818 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.439799 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientMemory" Jul 30 14:47:53.439885 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.439838 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 14:47:53.439885 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.439854 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientPID" Jul 30 14:47:53.440591 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.440574 2564 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 30 14:47:53.440591 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.440588 2564 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 30 14:47:53.440696 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.440606 2564 state_mem.go:36] "Initialized new in-memory state store" Jul 30 14:47:53.440734 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.440603 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 14:47:53.441984 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.441914 2564 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-38.ec2.internal.18c718a4d13cd5b4 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-38.ec2.internal,UID:ip-10-0-130-38.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-38.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-38.ec2.internal,},FirstTimestamp:2026-07-30 14:47:53.439819188 +0000 UTC m=+0.423102523,LastTimestamp:2026-07-30 14:47:53.439819188 +0000 UTC m=+0.423102523,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-38.ec2.internal,}" Jul 30 14:47:53.442087 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.442054 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientMemory" Jul 30 14:47:53.442159 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.442128 2564 policy_none.go:49] "None policy: Start" Jul 30 14:47:53.442159 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.442145 2564 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 30 14:47:53.442159 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.442152 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 14:47:53.442283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.442159 2564 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 30 14:47:53.442283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.442169 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientPID" Jul 30 14:47:53.443239 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.443226 2564 policy_none.go:47] "Start" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.483488 2564 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.483685 2564 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.483709 2564 kubelet.go:2448] "Starting kubelet main sync loop" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.483744 2564 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.483944 2564 manager.go:337] "Starting Device Plugin manager" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.483987 2564 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.483999 2564 server.go:87] "Starting device plugin registration server" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.484349 2564 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.484364 2564 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.484547 2564 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.484637 2564 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.484648 2564 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.486540 2564 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.486592 2564 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:53.486729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.486625 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 14:47:53.583918 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.583823 2564 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal"] Jul 30 14:47:53.584044 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.583928 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 14:47:53.585353 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.585334 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 14:47:53.585735 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.585717 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientMemory" Jul 30 14:47:53.585850 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.585751 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 14:47:53.585850 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.585761 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientPID" Jul 30 14:47:53.586309 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.586287 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientMemory" Jul 30 14:47:53.586383 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.586323 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 14:47:53.586383 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.586337 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientPID" Jul 30 14:47:53.586383 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.586370 2564 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.587047 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.587033 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 14:47:53.587188 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.587173 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.587263 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.587206 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 14:47:53.588361 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.588344 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientMemory" Jul 30 14:47:53.588435 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.588375 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 14:47:53.588435 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.588385 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientMemory" Jul 30 14:47:53.588435 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.588390 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientPID" Jul 30 14:47:53.588435 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.588405 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 14:47:53.588435 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.588418 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientPID" Jul 30 14:47:53.589601 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.589584 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.589698 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.589615 2564 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 30 14:47:53.590341 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.590324 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientMemory" Jul 30 14:47:53.590437 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.590353 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasNoDiskPressure" Jul 30 14:47:53.590437 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.590363 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeHasSufficientPID" Jul 30 14:47:53.595349 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.595331 2564 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.595439 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.595353 2564 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-38.ec2.internal\": node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:53.618813 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.618787 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13e490c80b8a599b5848621b448f965c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal\" (UID: \"13e490c80b8a599b5848621b448f965c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.618966 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.618820 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13e490c80b8a599b5848621b448f965c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal\" (UID: \"13e490c80b8a599b5848621b448f965c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.618966 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.618839 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b654a953333e0c5fd335382e9fed72ea-config\") pod \"kube-apiserver-proxy-ip-10-0-130-38.ec2.internal\" (UID: \"b654a953333e0c5fd335382e9fed72ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.621293 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.621273 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:53.624995 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.624974 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-38.ec2.internal\" not found" node="ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.629730 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.629709 2564 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-38.ec2.internal\" not found" node="ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.719008 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.718978 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13e490c80b8a599b5848621b448f965c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal\" (UID: \"13e490c80b8a599b5848621b448f965c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.719008 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.719013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13e490c80b8a599b5848621b448f965c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal\" (UID: \"13e490c80b8a599b5848621b448f965c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.719192 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.719032 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b654a953333e0c5fd335382e9fed72ea-config\") pod \"kube-apiserver-proxy-ip-10-0-130-38.ec2.internal\" (UID: \"b654a953333e0c5fd335382e9fed72ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.719192 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.719098 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/13e490c80b8a599b5848621b448f965c-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal\" (UID: \"13e490c80b8a599b5848621b448f965c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.719192 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.719110 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/13e490c80b8a599b5848621b448f965c-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal\" (UID: \"13e490c80b8a599b5848621b448f965c\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.719192 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.719160 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b654a953333e0c5fd335382e9fed72ea-config\") pod \"kube-apiserver-proxy-ip-10-0-130-38.ec2.internal\" (UID: \"b654a953333e0c5fd335382e9fed72ea\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.722064 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.722044 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:53.822516 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.822482 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:53.923300 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:53.923226 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:53.927411 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.927394 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:53.933220 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:53.933201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" Jul 30 14:47:54.023957 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.023915 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:54.124434 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.124407 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:54.225428 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.225326 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:54.325877 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.325836 2564 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-38.ec2.internal\" not found" Jul 30 14:47:54.346296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.346264 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 14:47:54.353766 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.353736 2564 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 30 14:47:54.353925 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.353893 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 14:47:54.353967 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.353925 2564 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 30 14:47:54.394516 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.394484 2564 apiserver.go:52] "Watching apiserver" Jul 30 14:47:54.411884 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.411855 2564 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 30 14:47:54.413426 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.413401 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-p86cc","openshift-cluster-node-tuning-operator/tuned-nfjcl","openshift-dns/node-resolver-dhkrf","openshift-image-registry/node-ca-cxskg","openshift-multus/multus-additional-cni-plugins-bm8gg","openshift-multus/multus-zq4qg","openshift-network-diagnostics/network-check-target-gm8xd","kube-system/konnectivity-agent-8v5bk","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv","openshift-multus/network-metrics-daemon-4k86d","openshift-network-operator/iptables-alerter-gz4q8","openshift-ovn-kubernetes/ovnkube-node-6r57d"] Jul 30 14:47:54.415247 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.415228 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.415323 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.415304 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:47:54.416946 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.416921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.417238 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.417160 2564 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 30 14:47:54.417238 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.417213 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" Jul 30 14:47:54.419997 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.419172 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.419997 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.419263 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 30 14:47:54.419997 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.419508 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zdrxt\"" Jul 30 14:47:54.419997 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.419858 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 30 14:47:54.422656 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422372 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 30 14:47:54.422656 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422378 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 30 14:47:54.422656 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422485 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-lvf8r\"" Jul 30 14:47:54.422656 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422632 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0048d797-3821-4f51-92ed-84cb6b974158-dbus\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.423006 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422662 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.423006 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-modprobe-d\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423006 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422713 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.423006 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422714 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysctl-d\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423006 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-systemd\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423006 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-host\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423006 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422952 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r5sh\" (UniqueName: \"kubernetes.io/projected/36159941-b1b2-4d47-98ce-cf212bf196a2-kube-api-access-9r5sh\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.423006 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.422982 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-run\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423021 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-sys\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d06e4219-5859-4453-b229-363a19484ec1-tmp\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423113 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0048d797-3821-4f51-92ed-84cb6b974158-kubelet-config\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-kubernetes\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423186 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-var-lib-kubelet\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423218 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r42l6\" (UniqueName: \"kubernetes.io/projected/d06e4219-5859-4453-b229-363a19484ec1-kube-api-access-r42l6\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36159941-b1b2-4d47-98ce-cf212bf196a2-hosts-file\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36159941-b1b2-4d47-98ce-cf212bf196a2-tmp-dir\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysconfig\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423359 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysctl-conf\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-lib-modules\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.423441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.423416 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d06e4219-5859-4453-b229-363a19484ec1-etc-tuned\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.424600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.424578 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.424918 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.424896 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 30 14:47:54.425121 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.425072 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p97th\"" Jul 30 14:47:54.425121 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.425092 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 30 14:47:54.425304 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.425289 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 30 14:47:54.426681 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.426663 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.427013 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.426997 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 30 14:47:54.427286 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.427270 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 30 14:47:54.427395 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.427377 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 30 14:47:54.427465 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.427382 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 30 14:47:54.427524 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.427485 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 30 14:47:54.427580 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.427550 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hpwxn\"" Jul 30 14:47:54.427698 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.427678 2564 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 30 14:47:54.428433 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.428418 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 30 14:47:54.428675 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.428660 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nggbp\"" Jul 30 14:47:54.429460 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.429443 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:47:54.429562 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.429539 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:47:54.431789 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.431770 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:47:54.433680 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.433656 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 14:47:54.433804 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.433759 2564 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" Jul 30 14:47:54.434091 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.434062 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-4cdzm\"" Jul 30 14:47:54.434358 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.434343 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 30 14:47:54.434395 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.434366 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 30 14:47:54.434706 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.434690 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.436665 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.436646 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 30 14:47:54.437015 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.436938 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:54.437183 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.437153 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:47:54.437301 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.437241 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 30 14:47:54.437351 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.437332 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jvgs6\"" Jul 30 14:47:54.437387 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.437371 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 30 14:47:54.439902 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.439873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.441843 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.441807 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-29 14:42:53 +0000 UTC" deadline="2028-04-15 22:07:35.635024816 +0000 UTC" Jul 30 14:47:54.441934 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.441848 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15007h19m41.193183835s" Jul 30 14:47:54.441934 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.441871 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 30 14:47:54.442018 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.441972 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 30 14:47:54.442105 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.442077 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 30 14:47:54.442212 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.442198 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-b7j5d\"" Jul 30 14:47:54.442474 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.442458 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal"] Jul 30 14:47:54.442632 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.442611 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.443184 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.443164 2564 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 30 14:47:54.443523 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.443508 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal"] Jul 30 14:47:54.444617 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.444597 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 30 14:47:54.444748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.444732 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 30 14:47:54.445067 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.445053 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 30 14:47:54.445179 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.445149 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 30 14:47:54.445489 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.445473 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 30 14:47:54.445569 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.445486 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tl2xx\"" Jul 30 14:47:54.445716 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.445699 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 30 14:47:54.465312 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.465286 2564 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-mdmjz" Jul 30 14:47:54.475080 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.475052 2564 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-mdmjz" Jul 30 14:47:54.505539 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.505509 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13e490c80b8a599b5848621b448f965c.slice/crio-0fe353cd7539dc785fda36c0a7ae9d10f69be6303df9d8378b927ff740db91f1 WatchSource:0}: Error finding container 0fe353cd7539dc785fda36c0a7ae9d10f69be6303df9d8378b927ff740db91f1: Status 404 returned error can't find the container with id 0fe353cd7539dc785fda36c0a7ae9d10f69be6303df9d8378b927ff740db91f1 Jul 30 14:47:54.505765 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.505751 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb654a953333e0c5fd335382e9fed72ea.slice/crio-b5a43fe8057403a6b0afefc50d46c05adfbf6bf3704916b0fbcfea5ef9d3d480 WatchSource:0}: Error finding container b5a43fe8057403a6b0afefc50d46c05adfbf6bf3704916b0fbcfea5ef9d3d480: Status 404 returned error can't find the container with id b5a43fe8057403a6b0afefc50d46c05adfbf6bf3704916b0fbcfea5ef9d3d480 Jul 30 14:47:54.512114 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.512091 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 14:47:54.518866 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.518848 2564 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 30 14:47:54.523608 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523582 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-log-socket\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.523745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523653 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.523745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523704 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-lib-modules\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.523745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d06e4219-5859-4453-b229-363a19484ec1-etc-tuned\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.523920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523764 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftmcr\" (UniqueName: \"kubernetes.io/projected/771dcee2-efa1-4503-9e75-16b904ed3df2-kube-api-access-ftmcr\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.523920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-os-release\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.523920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523829 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.523920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523858 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdbtf\" (UniqueName: \"kubernetes.io/projected/8d65a2db-722e-429c-8957-590325f6a9df-kube-api-access-bdbtf\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.523920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523889 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.523920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523905 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-lib-modules\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.523920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523915 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/771dcee2-efa1-4503-9e75-16b904ed3df2-host\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523943 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-sys\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.523976 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524042 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-sys\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524044 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-k8s-cni-cncf-io\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.524180 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-multus-certs\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.524261 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret podName:0048d797-3821-4f51-92ed-84cb6b974158 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:55.024235456 +0000 UTC m=+2.007518774 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret") pod "global-pull-secret-syncer-p86cc" (UID: "0048d797-3821-4f51-92ed-84cb6b974158") : object "kube-system"/"original-pull-secret" not registered Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524292 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf992175-6c90-49cd-ae87-1f07a3ecdcef-agent-certs\") pod \"konnectivity-agent-8v5bk\" (UID: \"cf992175-6c90-49cd-ae87-1f07a3ecdcef\") " pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524322 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524348 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b76f0754-2727-41b2-9656-5bbca581adde-ovn-node-metrics-cert\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.524396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524380 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-kubernetes\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524406 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-var-lib-kubelet\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524443 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r42l6\" (UniqueName: \"kubernetes.io/projected/d06e4219-5859-4453-b229-363a19484ec1-kube-api-access-r42l6\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524472 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524483 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-var-lib-kubelet\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524450 2564 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524521 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5k2gs\" (UniqueName: \"kubernetes.io/projected/22a81aec-43f6-4bba-8661-5b4785b398bc-kube-api-access-5k2gs\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524560 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-kubelet\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524486 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-kubernetes\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524598 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrj8\" (UniqueName: \"kubernetes.io/projected/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-kube-api-access-qbrj8\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524627 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-var-lib-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysctl-conf\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-socket-dir-parent\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524747 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-conf-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524773 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf992175-6c90-49cd-ae87-1f07a3ecdcef-konnectivity-ca\") pod \"konnectivity-agent-8v5bk\" (UID: \"cf992175-6c90-49cd-ae87-1f07a3ecdcef\") " pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524791 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-ovn\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524807 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-run-ovn-kubernetes\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.524955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524830 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-modprobe-d\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524830 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysctl-conf\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524856 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-host\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524877 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-kubelet\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524919 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-host\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524933 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-modprobe-d\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524941 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-node-log\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524966 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-cni-netd\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.524997 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525055 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-run\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525179 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d06e4219-5859-4453-b229-363a19484ec1-tmp\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-hostroot\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525237 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-systemd-units\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525264 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36159941-b1b2-4d47-98ce-cf212bf196a2-hosts-file\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/771dcee2-efa1-4503-9e75-16b904ed3df2-serviceca\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-run\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525314 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-netns\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525340 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-etc-kubernetes\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.526079 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/36159941-b1b2-4d47-98ce-cf212bf196a2-hosts-file\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-etc-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525398 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysconfig\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525424 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-cni-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysconfig\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-os-release\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzdf9\" (UniqueName: \"kubernetes.io/projected/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-kube-api-access-gzdf9\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525571 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-ovnkube-config\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525592 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysctl-d\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525618 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-daemon-config\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525654 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-sys-fs\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525690 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-cni-multus\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-run-netns\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525733 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-sysctl-d\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525741 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-env-overrides\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525795 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-ovnkube-script-lib\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.527127 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525825 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525854 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525882 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525908 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-slash\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-system-cni-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525971 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-socket-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.525996 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d65a2db-722e-429c-8957-590325f6a9df-iptables-alerter-script\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526039 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-cni-bin\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526064 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjcvg\" (UniqueName: \"kubernetes.io/projected/b76f0754-2727-41b2-9656-5bbca581adde-kube-api-access-xjcvg\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526105 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0048d797-3821-4f51-92ed-84cb6b974158-dbus\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-systemd\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r5sh\" (UniqueName: \"kubernetes.io/projected/36159941-b1b2-4d47-98ce-cf212bf196a2-kube-api-access-9r5sh\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526178 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-registration-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526207 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj9nt\" (UniqueName: \"kubernetes.io/projected/5924df4e-6705-4f80-a583-6f2b5d021254-kube-api-access-vj9nt\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526230 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-cnibin\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526242 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d06e4219-5859-4453-b229-363a19484ec1-etc-systemd\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.528029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526252 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-cni-binary-copy\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526276 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-cni-bin\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526299 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0048d797-3821-4f51-92ed-84cb6b974158-kubelet-config\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526354 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36159941-b1b2-4d47-98ce-cf212bf196a2-tmp-dir\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-system-cni-dir\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526404 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-cnibin\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526437 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-device-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526478 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d65a2db-722e-429c-8957-590325f6a9df-host-slash\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526503 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-systemd\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526545 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0048d797-3821-4f51-92ed-84cb6b974158-dbus\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0048d797-3821-4f51-92ed-84cb6b974158-kubelet-config\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.526749 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36159941-b1b2-4d47-98ce-cf212bf196a2-tmp-dir\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.528676 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d06e4219-5859-4453-b229-363a19484ec1-tmp\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.528911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.528740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d06e4219-5859-4453-b229-363a19484ec1-etc-tuned\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.533195 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.533128 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r42l6\" (UniqueName: \"kubernetes.io/projected/d06e4219-5859-4453-b229-363a19484ec1-kube-api-access-r42l6\") pod \"tuned-nfjcl\" (UID: \"d06e4219-5859-4453-b229-363a19484ec1\") " pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.533743 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.533725 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r5sh\" (UniqueName: \"kubernetes.io/projected/36159941-b1b2-4d47-98ce-cf212bf196a2-kube-api-access-9r5sh\") pod \"node-resolver-dhkrf\" (UID: \"36159941-b1b2-4d47-98ce-cf212bf196a2\") " pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.627414 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.627414 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627428 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627451 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-slash\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627482 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-system-cni-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-socket-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627528 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d65a2db-722e-429c-8957-590325f6a9df-iptables-alerter-script\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-cni-bin\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627556 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-slash\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjcvg\" (UniqueName: \"kubernetes.io/projected/b76f0754-2727-41b2-9656-5bbca581adde-kube-api-access-xjcvg\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627612 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-system-cni-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627624 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-registration-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627655 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj9nt\" (UniqueName: \"kubernetes.io/projected/5924df4e-6705-4f80-a583-6f2b5d021254-kube-api-access-vj9nt\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627687 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-etc-selinux\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627722 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-registration-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627745 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-cni-bin\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.627745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627755 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-cnibin\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.628553 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627765 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-socket-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.628553 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627798 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-cni-binary-copy\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.628553 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627839 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-cnibin\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.628553 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.627951 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-cni-bin\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.628553 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628504 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8d65a2db-722e-429c-8957-590325f6a9df-iptables-alerter-script\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.628553 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628567 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628583 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-system-cni-dir\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628627 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-cnibin\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628578 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-device-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628716 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-cni-bin\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d65a2db-722e-429c-8957-590325f6a9df-host-slash\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628736 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-device-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-systemd\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-cnibin\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-log-socket\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628824 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-system-cni-dir\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628844 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-systemd\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.628882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-cni-binary-copy\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628924 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628932 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-log-socket\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-kubelet-dir\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.628983 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8d65a2db-722e-429c-8957-590325f6a9df-host-slash\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftmcr\" (UniqueName: \"kubernetes.io/projected/771dcee2-efa1-4503-9e75-16b904ed3df2-kube-api-access-ftmcr\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629183 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-os-release\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629352 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-os-release\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629384 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629449 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdbtf\" (UniqueName: \"kubernetes.io/projected/8d65a2db-722e-429c-8957-590325f6a9df-kube-api-access-bdbtf\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.629600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629539 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/771dcee2-efa1-4503-9e75-16b904ed3df2-host\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.630073 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629593 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.630073 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-k8s-cni-cncf-io\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630073 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629678 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-multus-certs\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630073 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629709 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/771dcee2-efa1-4503-9e75-16b904ed3df2-host\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.630073 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629723 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf992175-6c90-49cd-ae87-1f07a3ecdcef-agent-certs\") pod \"konnectivity-agent-8v5bk\" (UID: \"cf992175-6c90-49cd-ae87-1f07a3ecdcef\") " pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:47:54.630073 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629972 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-cni-binary-copy\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.630073 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.629984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-k8s-cni-cncf-io\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630073 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630072 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-multus-certs\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630172 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b76f0754-2727-41b2-9656-5bbca581adde-ovn-node-metrics-cert\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630238 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5k2gs\" (UniqueName: \"kubernetes.io/projected/22a81aec-43f6-4bba-8661-5b4785b398bc-kube-api-access-5k2gs\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630271 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-kubelet\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630305 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrj8\" (UniqueName: \"kubernetes.io/projected/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-kube-api-access-qbrj8\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630339 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-var-lib-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630372 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-socket-dir-parent\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-conf-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630478 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf992175-6c90-49cd-ae87-1f07a3ecdcef-konnectivity-ca\") pod \"konnectivity-agent-8v5bk\" (UID: \"cf992175-6c90-49cd-ae87-1f07a3ecdcef\") " pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630527 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-ovn\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-run-ovn-kubernetes\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630583 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/22a81aec-43f6-4bba-8661-5b4785b398bc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630735 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-kubelet\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630764 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-node-log\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630794 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-cni-netd\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630894 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630953 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-hostroot\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.630995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.630984 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-systemd-units\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631013 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/771dcee2-efa1-4503-9e75-16b904ed3df2-serviceca\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631070 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-netns\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.631084 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-etc-kubernetes\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631166 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-etc-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631195 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-var-lib-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631203 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-cni-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631266 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-os-release\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631279 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-socket-dir-parent\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631304 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzdf9\" (UniqueName: \"kubernetes.io/projected/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-kube-api-access-gzdf9\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-conf-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.631341 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs podName:bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:55.131293173 +0000 UTC m=+2.114576508 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs") pod "network-metrics-daemon-4k86d" (UID: "bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631447 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-run-ovn-kubernetes\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.631512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631514 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-kubelet\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.632211 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.631923 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/cf992175-6c90-49cd-ae87-1f07a3ecdcef-konnectivity-ca\") pod \"konnectivity-agent-8v5bk\" (UID: \"cf992175-6c90-49cd-ae87-1f07a3ecdcef\") " pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:47:54.633083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.632553 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-ovnkube-config\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.632601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-daemon-config\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.633083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.632656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-sys-fs\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.633083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.632691 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-cni-multus\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.633083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.632729 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-run-netns\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.632763 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-env-overrides\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.632797 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-ovnkube-script-lib\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633036 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/5924df4e-6705-4f80-a583-6f2b5d021254-sys-fs\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633184 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-kubelet\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-var-lib-cni-multus\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-cni-netd\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633296 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-node-log\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633518 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-os-release\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633351 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-hostroot\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633355 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-systemd-units\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633312 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633417 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-host-run-netns\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633478 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-etc-kubernetes\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.633599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-etc-openvswitch\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.634234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633682 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-ovnkube-script-lib\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.634234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633680 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/22a81aec-43f6-4bba-8661-5b4785b398bc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.634234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.633766 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-cni-dir\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.634234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.634148 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/771dcee2-efa1-4503-9e75-16b904ed3df2-serviceca\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.634234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.634178 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-ovnkube-config\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.634469 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.634241 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 14:47:54.634469 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.634254 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/cf992175-6c90-49cd-ae87-1f07a3ecdcef-agent-certs\") pod \"konnectivity-agent-8v5bk\" (UID: \"cf992175-6c90-49cd-ae87-1f07a3ecdcef\") " pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:47:54.634469 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.634264 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 14:47:54.634469 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.634312 2564 projected.go:196] Error preparing data for projected volume kube-api-access-scpv7 for pod openshift-network-diagnostics/network-check-target-gm8xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:47:54.634469 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.634392 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-multus-daemon-config\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.635124 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.634479 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-run-ovn\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.635124 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.634394 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b76f0754-2727-41b2-9656-5bbca581adde-env-overrides\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.635124 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.634779 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b76f0754-2727-41b2-9656-5bbca581adde-host-run-netns\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.635124 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:54.634806 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7 podName:e4815781-a404-4160-b0e6-dd4f866d0ca4 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:55.134379051 +0000 UTC m=+2.117662389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-scpv7" (UniqueName: "kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7") pod "network-check-target-gm8xd" (UID: "e4815781-a404-4160-b0e6-dd4f866d0ca4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:47:54.637381 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.637357 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj9nt\" (UniqueName: \"kubernetes.io/projected/5924df4e-6705-4f80-a583-6f2b5d021254-kube-api-access-vj9nt\") pod \"aws-ebs-csi-driver-node-pvxgv\" (UID: \"5924df4e-6705-4f80-a583-6f2b5d021254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.637732 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.637606 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b76f0754-2727-41b2-9656-5bbca581adde-ovn-node-metrics-cert\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.637732 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.637704 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjcvg\" (UniqueName: \"kubernetes.io/projected/b76f0754-2727-41b2-9656-5bbca581adde-kube-api-access-xjcvg\") pod \"ovnkube-node-6r57d\" (UID: \"b76f0754-2727-41b2-9656-5bbca581adde\") " pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.637912 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.637890 2564 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 14:47:54.639685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.639646 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftmcr\" (UniqueName: \"kubernetes.io/projected/771dcee2-efa1-4503-9e75-16b904ed3df2-kube-api-access-ftmcr\") pod \"node-ca-cxskg\" (UID: \"771dcee2-efa1-4503-9e75-16b904ed3df2\") " pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.639685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.639669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdbtf\" (UniqueName: \"kubernetes.io/projected/8d65a2db-722e-429c-8957-590325f6a9df-kube-api-access-bdbtf\") pod \"iptables-alerter-gz4q8\" (UID: \"8d65a2db-722e-429c-8957-590325f6a9df\") " pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.640835 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.640815 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzdf9\" (UniqueName: \"kubernetes.io/projected/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-kube-api-access-gzdf9\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:54.641689 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.641666 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrj8\" (UniqueName: \"kubernetes.io/projected/7f7a7005-12ab-4a83-b8d7-4ea32d1cc705-kube-api-access-qbrj8\") pod \"multus-zq4qg\" (UID: \"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705\") " pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.642258 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.642240 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5k2gs\" (UniqueName: \"kubernetes.io/projected/22a81aec-43f6-4bba-8661-5b4785b398bc-kube-api-access-5k2gs\") pod \"multus-additional-cni-plugins-bm8gg\" (UID: \"22a81aec-43f6-4bba-8661-5b4785b398bc\") " pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.749302 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.749270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" Jul 30 14:47:54.760982 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.760957 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd06e4219_5859_4453_b229_363a19484ec1.slice/crio-21025f0a81e3aa5bb5698be044554272893bb1892d45893c686398ed29727605 WatchSource:0}: Error finding container 21025f0a81e3aa5bb5698be044554272893bb1892d45893c686398ed29727605: Status 404 returned error can't find the container with id 21025f0a81e3aa5bb5698be044554272893bb1892d45893c686398ed29727605 Jul 30 14:47:54.767314 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.767292 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dhkrf" Jul 30 14:47:54.774393 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.774355 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cxskg" Jul 30 14:47:54.775961 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.775937 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36159941_b1b2_4d47_98ce_cf212bf196a2.slice/crio-b9b50bb1755be5cfcad8a9458673a3eaf909e9999ea0ed534aea1011dbb7e84b WatchSource:0}: Error finding container b9b50bb1755be5cfcad8a9458673a3eaf909e9999ea0ed534aea1011dbb7e84b: Status 404 returned error can't find the container with id b9b50bb1755be5cfcad8a9458673a3eaf909e9999ea0ed534aea1011dbb7e84b Jul 30 14:47:54.783517 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.783491 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771dcee2_efa1_4503_9e75_16b904ed3df2.slice/crio-9be1b75a1016399ab3637a7d4b6b9c7c5f53f322d07e6fc2f7c552676afbd328 WatchSource:0}: Error finding container 9be1b75a1016399ab3637a7d4b6b9c7c5f53f322d07e6fc2f7c552676afbd328: Status 404 returned error can't find the container with id 9be1b75a1016399ab3637a7d4b6b9c7c5f53f322d07e6fc2f7c552676afbd328 Jul 30 14:47:54.793521 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.793498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" Jul 30 14:47:54.802658 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.802632 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22a81aec_43f6_4bba_8661_5b4785b398bc.slice/crio-1c30251aa64c6705acee6ae81a28e358a90ad9f29a549c21826687f9324aa171 WatchSource:0}: Error finding container 1c30251aa64c6705acee6ae81a28e358a90ad9f29a549c21826687f9324aa171: Status 404 returned error can't find the container with id 1c30251aa64c6705acee6ae81a28e358a90ad9f29a549c21826687f9324aa171 Jul 30 14:47:54.807012 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.806996 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-zq4qg" Jul 30 14:47:54.815532 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.815508 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f7a7005_12ab_4a83_b8d7_4ea32d1cc705.slice/crio-13cdb338e9e0d157e556ff99f0d030676f9c585441cbbf4703e29acb6e3c4810 WatchSource:0}: Error finding container 13cdb338e9e0d157e556ff99f0d030676f9c585441cbbf4703e29acb6e3c4810: Status 404 returned error can't find the container with id 13cdb338e9e0d157e556ff99f0d030676f9c585441cbbf4703e29acb6e3c4810 Jul 30 14:47:54.820668 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.820651 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:47:54.827728 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.827704 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" Jul 30 14:47:54.828937 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.828914 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf992175_6c90_49cd_ae87_1f07a3ecdcef.slice/crio-aaae7008076dc127c5b611640b52474271e3ade20861a8f69e88399dde242954 WatchSource:0}: Error finding container aaae7008076dc127c5b611640b52474271e3ade20861a8f69e88399dde242954: Status 404 returned error can't find the container with id aaae7008076dc127c5b611640b52474271e3ade20861a8f69e88399dde242954 Jul 30 14:47:54.836613 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.836585 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5924df4e_6705_4f80_a583_6f2b5d021254.slice/crio-7d39db26428ed616b6267bfe0d45399e2d933d10badf91c889c5ad132eb2c44e WatchSource:0}: Error finding container 7d39db26428ed616b6267bfe0d45399e2d933d10badf91c889c5ad132eb2c44e: Status 404 returned error can't find the container with id 7d39db26428ed616b6267bfe0d45399e2d933d10badf91c889c5ad132eb2c44e Jul 30 14:47:54.844317 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.844298 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gz4q8" Jul 30 14:47:54.849470 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:54.849447 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:47:54.854689 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.854665 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d65a2db_722e_429c_8957_590325f6a9df.slice/crio-94d53dbc65c24ef354a0bc21807f0590201b8c87a97a960aaf48d298ace9dfa3 WatchSource:0}: Error finding container 94d53dbc65c24ef354a0bc21807f0590201b8c87a97a960aaf48d298ace9dfa3: Status 404 returned error can't find the container with id 94d53dbc65c24ef354a0bc21807f0590201b8c87a97a960aaf48d298ace9dfa3 Jul 30 14:47:54.858802 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:47:54.858774 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76f0754_2727_41b2_9656_5bbca581adde.slice/crio-bcb6f471b2a7a397d2ae165259a516bf078e9b0f0a55b3e690d0a1aaae8c468c WatchSource:0}: Error finding container bcb6f471b2a7a397d2ae165259a516bf078e9b0f0a55b3e690d0a1aaae8c468c: Status 404 returned error can't find the container with id bcb6f471b2a7a397d2ae165259a516bf078e9b0f0a55b3e690d0a1aaae8c468c Jul 30 14:47:55.013405 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.013309 2564 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 14:47:55.037025 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.036983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:55.037221 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:55.037177 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 14:47:55.037278 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:55.037249 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret podName:0048d797-3821-4f51-92ed-84cb6b974158 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:56.037233046 +0000 UTC m=+3.020516362 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret") pod "global-pull-secret-syncer-p86cc" (UID: "0048d797-3821-4f51-92ed-84cb6b974158") : object "kube-system"/"original-pull-secret" not registered Jul 30 14:47:55.138185 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.138141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:55.138388 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.138223 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:47:55.138388 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:55.138316 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:47:55.138388 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:55.138384 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 14:47:55.138550 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:55.138398 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs podName:bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:56.138381181 +0000 UTC m=+3.121664515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs") pod "network-metrics-daemon-4k86d" (UID: "bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:47:55.138550 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:55.138400 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 14:47:55.138550 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:55.138423 2564 projected.go:196] Error preparing data for projected volume kube-api-access-scpv7 for pod openshift-network-diagnostics/network-check-target-gm8xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:47:55.138550 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:55.138461 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7 podName:e4815781-a404-4160-b0e6-dd4f866d0ca4 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:56.138452793 +0000 UTC m=+3.121736146 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-scpv7" (UniqueName: "kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7") pod "network-check-target-gm8xd" (UID: "e4815781-a404-4160-b0e6-dd4f866d0ca4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:47:55.478038 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.477965 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 14:42:54 +0000 UTC" deadline="2028-01-19 23:18:09.984515148 +0000 UTC" Jul 30 14:47:55.478038 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.478039 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12920h30m14.506480656s" Jul 30 14:47:55.508334 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.508196 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" event={"ID":"5924df4e-6705-4f80-a583-6f2b5d021254","Type":"ContainerStarted","Data":"7d39db26428ed616b6267bfe0d45399e2d933d10badf91c889c5ad132eb2c44e"} Jul 30 14:47:55.519760 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.519648 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zq4qg" event={"ID":"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705","Type":"ContainerStarted","Data":"13cdb338e9e0d157e556ff99f0d030676f9c585441cbbf4703e29acb6e3c4810"} Jul 30 14:47:55.527497 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.527422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" event={"ID":"22a81aec-43f6-4bba-8661-5b4785b398bc","Type":"ContainerStarted","Data":"1c30251aa64c6705acee6ae81a28e358a90ad9f29a549c21826687f9324aa171"} Jul 30 14:47:55.533394 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.533355 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dhkrf" event={"ID":"36159941-b1b2-4d47-98ce-cf212bf196a2","Type":"ContainerStarted","Data":"b9b50bb1755be5cfcad8a9458673a3eaf909e9999ea0ed534aea1011dbb7e84b"} Jul 30 14:47:55.545246 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.545201 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" event={"ID":"d06e4219-5859-4453-b229-363a19484ec1","Type":"ContainerStarted","Data":"21025f0a81e3aa5bb5698be044554272893bb1892d45893c686398ed29727605"} Jul 30 14:47:55.553905 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.553808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" event={"ID":"b654a953333e0c5fd335382e9fed72ea","Type":"ContainerStarted","Data":"b5a43fe8057403a6b0afefc50d46c05adfbf6bf3704916b0fbcfea5ef9d3d480"} Jul 30 14:47:55.557606 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.557530 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8v5bk" event={"ID":"cf992175-6c90-49cd-ae87-1f07a3ecdcef","Type":"ContainerStarted","Data":"aaae7008076dc127c5b611640b52474271e3ade20861a8f69e88399dde242954"} Jul 30 14:47:55.569138 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.569094 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxskg" event={"ID":"771dcee2-efa1-4503-9e75-16b904ed3df2","Type":"ContainerStarted","Data":"9be1b75a1016399ab3637a7d4b6b9c7c5f53f322d07e6fc2f7c552676afbd328"} Jul 30 14:47:55.577889 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.577850 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" event={"ID":"13e490c80b8a599b5848621b448f965c","Type":"ContainerStarted","Data":"0fe353cd7539dc785fda36c0a7ae9d10f69be6303df9d8378b927ff740db91f1"} Jul 30 14:47:55.583412 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.583370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"bcb6f471b2a7a397d2ae165259a516bf078e9b0f0a55b3e690d0a1aaae8c468c"} Jul 30 14:47:55.590780 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.590742 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gz4q8" event={"ID":"8d65a2db-722e-429c-8957-590325f6a9df","Type":"ContainerStarted","Data":"94d53dbc65c24ef354a0bc21807f0590201b8c87a97a960aaf48d298ace9dfa3"} Jul 30 14:47:55.671351 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:55.671319 2564 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 14:47:56.046991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.046944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:56.047200 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.047147 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 14:47:56.047286 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.047223 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret podName:0048d797-3821-4f51-92ed-84cb6b974158 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:58.047201947 +0000 UTC m=+5.030485270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret") pod "global-pull-secret-syncer-p86cc" (UID: "0048d797-3821-4f51-92ed-84cb6b974158") : object "kube-system"/"original-pull-secret" not registered Jul 30 14:47:56.148220 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.148163 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:56.148517 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.148250 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:47:56.148517 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.148439 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 14:47:56.148517 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.148461 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 14:47:56.148517 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.148475 2564 projected.go:196] Error preparing data for projected volume kube-api-access-scpv7 for pod openshift-network-diagnostics/network-check-target-gm8xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:47:56.148834 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.148540 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7 podName:e4815781-a404-4160-b0e6-dd4f866d0ca4 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:58.148518383 +0000 UTC m=+5.131801718 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-scpv7" (UniqueName: "kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7") pod "network-check-target-gm8xd" (UID: "e4815781-a404-4160-b0e6-dd4f866d0ca4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:47:56.149087 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.148997 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:47:56.149087 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.149054 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs podName:bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71 nodeName:}" failed. No retries permitted until 2026-07-30 14:47:58.149038467 +0000 UTC m=+5.132321802 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs") pod "network-metrics-daemon-4k86d" (UID: "bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:47:56.478979 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.478933 2564 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-29 14:42:54 +0000 UTC" deadline="2028-04-21 12:52:30.402844788 +0000 UTC" Jul 30 14:47:56.478979 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.478979 2564 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15142h4m33.923870834s" Jul 30 14:47:56.484536 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.484512 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:56.484695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.484553 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:47:56.484695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.484554 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:56.484695 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.484673 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:47:56.484869 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.484789 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:47:56.484931 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:56.484863 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:47:56.842058 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:56.841976 2564 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 30 14:47:58.067909 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:58.067860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:58.068494 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.068049 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 14:47:58.068494 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.068141 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret podName:0048d797-3821-4f51-92ed-84cb6b974158 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:02.068102562 +0000 UTC m=+9.051385901 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret") pod "global-pull-secret-syncer-p86cc" (UID: "0048d797-3821-4f51-92ed-84cb6b974158") : object "kube-system"/"original-pull-secret" not registered Jul 30 14:47:58.169299 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:58.169244 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:47:58.169478 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:58.169343 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:58.169576 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.169508 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:47:58.169639 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.169580 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs podName:bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:02.169558866 +0000 UTC m=+9.152842188 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs") pod "network-metrics-daemon-4k86d" (UID: "bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:47:58.170234 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.170019 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 14:47:58.170234 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.170046 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 14:47:58.170234 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.170059 2564 projected.go:196] Error preparing data for projected volume kube-api-access-scpv7 for pod openshift-network-diagnostics/network-check-target-gm8xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:47:58.170234 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.170106 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7 podName:e4815781-a404-4160-b0e6-dd4f866d0ca4 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:02.170089912 +0000 UTC m=+9.153373235 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-scpv7" (UniqueName: "kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7") pod "network-check-target-gm8xd" (UID: "e4815781-a404-4160-b0e6-dd4f866d0ca4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:47:58.484944 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:58.484899 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:47:58.485078 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.485049 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:47:58.485256 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:58.485240 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:47:58.485697 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.485352 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:47:58.485697 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:47:58.485432 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:47:58.485697 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:47:58.485533 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:00.484600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:00.484568 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:00.485037 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:00.484610 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:00.485037 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:00.484716 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:00.485037 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:00.484774 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:00.485037 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:00.484828 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:00.485037 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:00.484913 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:02.106614 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:02.106314 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:02.106614 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.106526 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 14:48:02.106614 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.106592 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret podName:0048d797-3821-4f51-92ed-84cb6b974158 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:10.106571952 +0000 UTC m=+17.089855288 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret") pod "global-pull-secret-syncer-p86cc" (UID: "0048d797-3821-4f51-92ed-84cb6b974158") : object "kube-system"/"original-pull-secret" not registered Jul 30 14:48:02.207549 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:02.207512 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:02.207732 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:02.207581 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:02.207807 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.207785 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:48:02.207959 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.207852 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs podName:bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:10.207829981 +0000 UTC m=+17.191113319 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs") pod "network-metrics-daemon-4k86d" (UID: "bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:48:02.207959 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.207867 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 14:48:02.207959 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.207885 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 14:48:02.207959 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.207897 2564 projected.go:196] Error preparing data for projected volume kube-api-access-scpv7 for pod openshift-network-diagnostics/network-check-target-gm8xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:48:02.207959 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.207930 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7 podName:e4815781-a404-4160-b0e6-dd4f866d0ca4 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:10.20792041 +0000 UTC m=+17.191203741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-scpv7" (UniqueName: "kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7") pod "network-check-target-gm8xd" (UID: "e4815781-a404-4160-b0e6-dd4f866d0ca4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:48:02.484449 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:02.483929 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:02.484449 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:02.484097 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:02.484449 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.484103 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:02.484449 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:02.484134 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:02.484449 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.484222 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:02.484449 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:02.484328 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:04.484800 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:04.484502 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:04.484800 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:04.484532 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:04.484800 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:04.484646 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:04.485360 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:04.485003 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:04.485360 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:04.485142 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:04.485360 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:04.485231 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:06.484678 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:06.484642 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:06.485146 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:06.484642 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:06.485146 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:06.484642 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:06.485146 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:06.484770 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:06.485146 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:06.484805 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:06.485146 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:06.484873 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:08.484860 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:08.484821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:08.485325 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:08.484821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:08.485325 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:08.484969 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:08.485325 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:08.484821 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:08.485325 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:08.485022 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:08.485325 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:08.485141 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:10.163272 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:10.163226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:10.163745 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.163418 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 14:48:10.163745 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.163496 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret podName:0048d797-3821-4f51-92ed-84cb6b974158 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:26.163475118 +0000 UTC m=+33.146758435 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret") pod "global-pull-secret-syncer-p86cc" (UID: "0048d797-3821-4f51-92ed-84cb6b974158") : object "kube-system"/"original-pull-secret" not registered Jul 30 14:48:10.264317 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:10.264276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:10.264481 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:10.264355 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:10.264481 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.264450 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:48:10.264599 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.264512 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs podName:bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:26.264492351 +0000 UTC m=+33.247775669 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs") pod "network-metrics-daemon-4k86d" (UID: "bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:48:10.264599 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.264453 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 14:48:10.264599 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.264550 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 14:48:10.264599 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.264564 2564 projected.go:196] Error preparing data for projected volume kube-api-access-scpv7 for pod openshift-network-diagnostics/network-check-target-gm8xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:48:10.264837 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.264614 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7 podName:e4815781-a404-4160-b0e6-dd4f866d0ca4 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:26.264601908 +0000 UTC m=+33.247885244 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-scpv7" (UniqueName: "kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7") pod "network-check-target-gm8xd" (UID: "e4815781-a404-4160-b0e6-dd4f866d0ca4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:48:10.484495 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:10.484459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:10.484663 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:10.484459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:10.484663 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.484588 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:10.484785 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.484685 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:10.484785 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:10.484459 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:10.484785 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:10.484779 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:12.484907 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:12.484867 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:12.485415 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:12.484871 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:12.485415 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:12.485007 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:12.485415 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:12.484871 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:12.485415 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:12.485113 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:12.485415 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:12.485202 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:14.484772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:14.484737 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:14.485164 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:14.484840 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:14.485164 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:14.484848 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:14.485164 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:14.484870 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:14.485164 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:14.484953 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:14.485164 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:14.485014 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:15.639616 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.639579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" event={"ID":"5924df4e-6705-4f80-a583-6f2b5d021254","Type":"ContainerStarted","Data":"6fa77dded4b62a945ca4f43464b947544b136cf490b50f4ddac7f52b42749fc5"} Jul 30 14:48:15.641410 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.641361 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-zq4qg" event={"ID":"7f7a7005-12ab-4a83-b8d7-4ea32d1cc705","Type":"ContainerStarted","Data":"256cb6ada2b684ea006dbec87c73e1ec6f0a4999141a784d7218c29645db4bbe"} Jul 30 14:48:15.643150 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.643121 2564 generic.go:356] "Generic (PLEG): container finished" podID="22a81aec-43f6-4bba-8661-5b4785b398bc" containerID="5a3ba4909a9c9f53bfd8ee88f5ba1d64c60b25e1b0096844b3608c572a07b545" exitCode=0 Jul 30 14:48:15.643266 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.643189 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" event={"ID":"22a81aec-43f6-4bba-8661-5b4785b398bc","Type":"ContainerDied","Data":"5a3ba4909a9c9f53bfd8ee88f5ba1d64c60b25e1b0096844b3608c572a07b545"} Jul 30 14:48:15.644670 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.644648 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dhkrf" event={"ID":"36159941-b1b2-4d47-98ce-cf212bf196a2","Type":"ContainerStarted","Data":"988bae23168fc596e832c020f0cd08237c565b2b5bc5e7c0f4ec45bce44d23dc"} Jul 30 14:48:15.646909 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.646847 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" event={"ID":"d06e4219-5859-4453-b229-363a19484ec1","Type":"ContainerStarted","Data":"586c1064983cb8b3cbf530d42d96c066c5c378f5741f87987f66cb45b5ca8c34"} Jul 30 14:48:15.648349 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.648330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" event={"ID":"b654a953333e0c5fd335382e9fed72ea","Type":"ContainerStarted","Data":"849331aa3dbe71f43d696dac7b709c0a2d52483ad42207bbd2f609dfc52d400c"} Jul 30 14:48:15.649698 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.649673 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-8v5bk" event={"ID":"cf992175-6c90-49cd-ae87-1f07a3ecdcef","Type":"ContainerStarted","Data":"a03a666ab92defe1a9e4e214662711f6a404188ccdd0e39ddfb475e6e073d646"} Jul 30 14:48:15.651219 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.651197 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cxskg" event={"ID":"771dcee2-efa1-4503-9e75-16b904ed3df2","Type":"ContainerStarted","Data":"8392a3821db39e64046d2f61b0786d94caf916c9f32ae93c650431a7a5132fea"} Jul 30 14:48:15.652669 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.652647 2564 generic.go:356] "Generic (PLEG): container finished" podID="13e490c80b8a599b5848621b448f965c" containerID="d4c325e0a7ffbc3ee685fd269c1a4fe58b42dadaf1907022914c54fb41c9b931" exitCode=0 Jul 30 14:48:15.652815 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.652717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" event={"ID":"13e490c80b8a599b5848621b448f965c","Type":"ContainerDied","Data":"d4c325e0a7ffbc3ee685fd269c1a4fe58b42dadaf1907022914c54fb41c9b931"} Jul 30 14:48:15.655471 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.655454 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 14:48:15.655797 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.655779 2564 generic.go:356] "Generic (PLEG): container finished" podID="b76f0754-2727-41b2-9656-5bbca581adde" containerID="91eea8b038f35cbc7574cf6f4c5e9449b7bf4f3b66e018854e62e23c4e4e70f3" exitCode=1 Jul 30 14:48:15.655857 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.655808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"1e4c17a07c79ffcf4aa49b60855270ed39e427096c5677cdc687aec7ca55122d"} Jul 30 14:48:15.655897 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.655865 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"33d3a4e8cba3cac2ca170245896173dde87c4a19554bc9028edccca1777ebf41"} Jul 30 14:48:15.655934 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.655901 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"b082932a852912b11e99d1ee514a64546ed4e12bf5811289ba569056678433fb"} Jul 30 14:48:15.655934 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.655929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"5834f5eb80f2913442893e90dfd49ffae58dd885d9913c89361798d0a160360c"} Jul 30 14:48:15.656003 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.655960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerDied","Data":"91eea8b038f35cbc7574cf6f4c5e9449b7bf4f3b66e018854e62e23c4e4e70f3"} Jul 30 14:48:15.656003 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.655989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"ade671159dc40b1b6263e70aa9a0b744cb8d4269841d39427c3b1c546c6046ec"} Jul 30 14:48:15.661068 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.661028 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-zq4qg" podStartSLOduration=1.25954356 podStartE2EDuration="21.661016358s" totalImagesPullingTime="20.401472798s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:15.660520167 +0000 UTC m=+22.643803516" watchObservedRunningTime="2026-07-30 14:48:15.661016358 +0000 UTC m=+22.644299695" Jul 30 14:48:15.676507 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.676464 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dhkrf" podStartSLOduration=1.5544817819999999 podStartE2EDuration="21.67644971s" totalImagesPullingTime="20.121967928s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:15.676424232 +0000 UTC m=+22.659707569" watchObservedRunningTime="2026-07-30 14:48:15.67644971 +0000 UTC m=+22.659733048" Jul 30 14:48:15.712224 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.712166 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-8v5bk" podStartSLOduration=1.655974163 podStartE2EDuration="21.712146797s" totalImagesPullingTime="20.056172634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:15.711620272 +0000 UTC m=+22.694903610" watchObservedRunningTime="2026-07-30 14:48:15.712146797 +0000 UTC m=+22.695430136" Jul 30 14:48:15.725091 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.725027 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-38.ec2.internal" podStartSLOduration=21.725011429 podStartE2EDuration="21.725011429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:15.724442225 +0000 UTC m=+22.707725563" watchObservedRunningTime="2026-07-30 14:48:15.725011429 +0000 UTC m=+22.708294766" Jul 30 14:48:15.770336 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.770286 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nfjcl" podStartSLOduration=1.628564337 podStartE2EDuration="21.770268188s" totalImagesPullingTime="20.141703851s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:15.744255373 +0000 UTC m=+22.727538709" watchObservedRunningTime="2026-07-30 14:48:15.770268188 +0000 UTC m=+22.753551547" Jul 30 14:48:15.770493 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:15.770413 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cxskg" podStartSLOduration=1.6554447890000001 podStartE2EDuration="21.770408632s" totalImagesPullingTime="20.114963843s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:15.769968044 +0000 UTC m=+22.753251380" watchObservedRunningTime="2026-07-30 14:48:15.770408632 +0000 UTC m=+22.753692011" Jul 30 14:48:16.484980 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:16.484785 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:16.485181 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:16.484854 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:16.485181 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:16.485112 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:16.485181 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:16.484855 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:16.485336 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:16.485188 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:16.485336 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:16.485275 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:16.659602 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:16.659509 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gz4q8" event={"ID":"8d65a2db-722e-429c-8957-590325f6a9df","Type":"ContainerStarted","Data":"e6ac3c201a9c1f5df12cf8d1498fd1730ed52e1fbcc777d67155d919f250a54f"} Jul 30 14:48:16.674352 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:16.674295 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gz4q8" podStartSLOduration=10.654341187 podStartE2EDuration="22.674274246s" totalImagesPullingTime="12.019933059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:16.673562782 +0000 UTC m=+23.656846118" watchObservedRunningTime="2026-07-30 14:48:16.674274246 +0000 UTC m=+23.657557584" Jul 30 14:48:16.858745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:16.858713 2564 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 30 14:48:17.498215 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:17.498119 2564 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-30T14:48:16.858740372Z","UUID":"cb1d1348-71c7-4f2d-b840-b699425d0720","Handler":null,"Name":"","Endpoint":""} Jul 30 14:48:17.499988 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:17.499955 2564 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 30 14:48:17.500150 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:17.499999 2564 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 30 14:48:17.665092 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:17.665039 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" event={"ID":"13e490c80b8a599b5848621b448f965c","Type":"ContainerStarted","Data":"138af33631d4b2d99e12e848cd2491c17fbae071cffa2c10f7972b10d61a587a"} Jul 30 14:48:17.667669 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:17.667638 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" event={"ID":"5924df4e-6705-4f80-a583-6f2b5d021254","Type":"ContainerStarted","Data":"fc4ffddabf364ca16c4bec4470f02130cd3cc18e53893fb000c2e85ae6f6f110"} Jul 30 14:48:17.681288 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:17.681228 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-38.ec2.internal" podStartSLOduration=23.681208823 podStartE2EDuration="23.681208823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:17.680511689 +0000 UTC m=+24.663795026" watchObservedRunningTime="2026-07-30 14:48:17.681208823 +0000 UTC m=+24.664492162" Jul 30 14:48:18.484139 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:18.484094 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:18.484139 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:18.484139 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:18.484368 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:18.484139 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:18.484368 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:18.484244 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:18.484368 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:18.484316 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:18.484465 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:18.484398 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:18.674303 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:18.674276 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 14:48:18.674775 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:18.674691 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"f9f47e95776015fca1330134e236aaf6d2652d8bf90e955515bc63ec23dc4943"} Jul 30 14:48:18.849478 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:18.849384 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:48:18.850134 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:18.850110 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:48:20.484282 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.484106 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:20.484955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.484112 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:20.484955 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:20.484360 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:20.484955 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:20.484445 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:20.484955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.484112 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:20.484955 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:20.484520 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:20.683020 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.682949 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 14:48:20.683356 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.683324 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"921a2c6110440eeb4690f1406762e2f4e6690638c5355ec2b3a122efdd66a2d4"} Jul 30 14:48:20.683569 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.683542 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:48:20.683781 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.683765 2564 scope.go:117] "RemoveContainer" containerID="91eea8b038f35cbc7574cf6f4c5e9449b7bf4f3b66e018854e62e23c4e4e70f3" Jul 30 14:48:20.685534 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.685511 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" event={"ID":"5924df4e-6705-4f80-a583-6f2b5d021254","Type":"ContainerStarted","Data":"247d42a1ed0453dd1d4ccbbd68d2a42dd76e8514a734ec697f67bdd759ed07c8"} Jul 30 14:48:20.687356 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.687332 2564 generic.go:356] "Generic (PLEG): container finished" podID="22a81aec-43f6-4bba-8661-5b4785b398bc" containerID="585f1d103f900075cf7bf5711af31cf9150f963875d8a759d4db558c0d935ae7" exitCode=0 Jul 30 14:48:20.687453 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.687372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" event={"ID":"22a81aec-43f6-4bba-8661-5b4785b398bc","Type":"ContainerDied","Data":"585f1d103f900075cf7bf5711af31cf9150f963875d8a759d4db558c0d935ae7"} Jul 30 14:48:20.700523 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.700503 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:48:20.780473 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:20.780421 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-pvxgv" podStartSLOduration=2.38453956 podStartE2EDuration="26.780406494s" totalImagesPullingTime="24.395866934s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:20.780256021 +0000 UTC m=+27.763539358" watchObservedRunningTime="2026-07-30 14:48:20.780406494 +0000 UTC m=+27.763689830" Jul 30 14:48:21.696101 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:21.695912 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 14:48:21.696561 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:21.696475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" event={"ID":"b76f0754-2727-41b2-9656-5bbca581adde","Type":"ContainerStarted","Data":"7d5524cb010104bd347b774e197f0d669e6e137173104f9075d35a77bfb60f41"} Jul 30 14:48:21.696883 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:21.696861 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:48:21.697025 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:21.696912 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:48:21.712197 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:21.712171 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:48:21.725598 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:21.725480 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" podStartSLOduration=7.655109345 podStartE2EDuration="27.725465331s" totalImagesPullingTime="20.070355986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:21.723925564 +0000 UTC m=+28.707208900" watchObservedRunningTime="2026-07-30 14:48:21.725465331 +0000 UTC m=+28.708748668" Jul 30 14:48:22.338957 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:22.338929 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p86cc"] Jul 30 14:48:22.339172 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:22.339061 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:22.339247 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:22.339172 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:22.341588 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:22.341562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gm8xd"] Jul 30 14:48:22.341714 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:22.341676 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:22.341786 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:22.341765 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:22.344420 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:22.344396 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4k86d"] Jul 30 14:48:22.344555 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:22.344514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:22.344641 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:22.344620 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:22.700698 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:22.700613 2564 generic.go:356] "Generic (PLEG): container finished" podID="22a81aec-43f6-4bba-8661-5b4785b398bc" containerID="6be4b6113f231673a8ebc0f442650e7e98641919cf8b4917172282b761a3a699" exitCode=0 Jul 30 14:48:22.701118 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:22.700699 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" event={"ID":"22a81aec-43f6-4bba-8661-5b4785b398bc","Type":"ContainerDied","Data":"6be4b6113f231673a8ebc0f442650e7e98641919cf8b4917172282b761a3a699"} Jul 30 14:48:23.485262 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:23.485232 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:23.485436 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:23.485325 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:24.484586 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:24.484409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:24.484962 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:24.484418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:24.484962 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:24.484658 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:24.484962 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:24.484723 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:24.708203 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:24.708122 2564 generic.go:356] "Generic (PLEG): container finished" podID="22a81aec-43f6-4bba-8661-5b4785b398bc" containerID="6e71fd3af11093454e7354aae53dcbb9d2ffe6a447a89a25f7a5382f41c24383" exitCode=0 Jul 30 14:48:24.708203 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:24.708170 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" event={"ID":"22a81aec-43f6-4bba-8661-5b4785b398bc","Type":"ContainerDied","Data":"6e71fd3af11093454e7354aae53dcbb9d2ffe6a447a89a25f7a5382f41c24383"} Jul 30 14:48:24.726080 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:24.726046 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:48:24.726230 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:24.726172 2564 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 30 14:48:24.726695 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:24.726678 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-8v5bk" Jul 30 14:48:25.484485 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:25.484449 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:25.484659 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:25.484586 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:26.185101 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:26.184987 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:26.185271 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.185158 2564 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 30 14:48:26.185271 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.185235 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret podName:0048d797-3821-4f51-92ed-84cb6b974158 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:58.185216643 +0000 UTC m=+65.168499960 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret") pod "global-pull-secret-syncer-p86cc" (UID: "0048d797-3821-4f51-92ed-84cb6b974158") : object "kube-system"/"original-pull-secret" not registered Jul 30 14:48:26.285451 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:26.285416 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:26.285682 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:26.285492 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:26.285682 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.285574 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:48:26.285682 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.285631 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 30 14:48:26.285682 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.285649 2564 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 30 14:48:26.285682 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.285660 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs podName:bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:58.285634266 +0000 UTC m=+65.268917588 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs") pod "network-metrics-daemon-4k86d" (UID: "bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 30 14:48:26.285682 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.285661 2564 projected.go:196] Error preparing data for projected volume kube-api-access-scpv7 for pod openshift-network-diagnostics/network-check-target-gm8xd: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:48:26.286090 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.285710 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7 podName:e4815781-a404-4160-b0e6-dd4f866d0ca4 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:58.285700136 +0000 UTC m=+65.268983465 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-scpv7" (UniqueName: "kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7") pod "network-check-target-gm8xd" (UID: "e4815781-a404-4160-b0e6-dd4f866d0ca4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 30 14:48:26.484863 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:26.484824 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:26.485332 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:26.484824 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:26.485332 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.484946 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gm8xd" podUID="e4815781-a404-4160-b0e6-dd4f866d0ca4" Jul 30 14:48:26.485332 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:26.485104 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p86cc" podUID="0048d797-3821-4f51-92ed-84cb6b974158" Jul 30 14:48:27.484843 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.484808 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:27.485095 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:27.484961 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4k86d" podUID="bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71" Jul 30 14:48:27.842117 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.842024 2564 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-38.ec2.internal" event="NodeReady" Jul 30 14:48:27.842268 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.842185 2564 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 30 14:48:27.878751 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.878716 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-757dcf8f9-8j2qt"] Jul 30 14:48:27.881446 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.881396 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:27.883878 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.883854 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 30 14:48:27.884268 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.884244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fxhsd\"" Jul 30 14:48:27.884385 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.884244 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 30 14:48:27.884385 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.884331 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 30 14:48:27.892888 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.892856 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 30 14:48:27.895677 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.895648 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-gx9sl"] Jul 30 14:48:27.897914 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.897893 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:27.899744 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.899650 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757dcf8f9-8j2qt"] Jul 30 14:48:27.900000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.899982 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 30 14:48:27.900470 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.900396 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 30 14:48:27.900470 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.900449 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2kjxw"] Jul 30 14:48:27.900638 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.900455 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vrbw2\"" Jul 30 14:48:27.903672 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.903653 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2kjxw" Jul 30 14:48:27.905611 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.905589 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 30 14:48:27.905854 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.905832 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 30 14:48:27.905854 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.905848 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 30 14:48:27.905994 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.905835 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-dcx9h\"" Jul 30 14:48:27.912258 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.912237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2kjxw"] Jul 30 14:48:27.916024 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.916002 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gx9sl"] Jul 30 14:48:27.997752 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.997719 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnlgj\" (UniqueName: \"kubernetes.io/projected/99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a-kube-api-access-wnlgj\") pod \"ingress-canary-2kjxw\" (UID: \"99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a\") " pod="openshift-ingress-canary/ingress-canary-2kjxw" Jul 30 14:48:27.997937 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.997766 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-image-registry-private-configuration\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:27.997937 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.997796 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:27.997937 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.997819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-certificates\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:27.997937 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.997841 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-installation-pull-secrets\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:27.997937 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.997864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4253f459-2c3c-427f-b9da-3449dbd7212f-ca-trust-extracted\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:27.998194 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.997932 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e334195-bba1-46db-8e7b-b746dbb85ea8-tmp-dir\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:27.998194 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.998063 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7fft\" (UniqueName: \"kubernetes.io/projected/3e334195-bba1-46db-8e7b-b746dbb85ea8-kube-api-access-d7fft\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:27.998194 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.998111 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a-cert\") pod \"ingress-canary-2kjxw\" (UID: \"99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a\") " pod="openshift-ingress-canary/ingress-canary-2kjxw" Jul 30 14:48:27.998194 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.998167 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-bound-sa-token\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:27.998194 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.998193 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e334195-bba1-46db-8e7b-b746dbb85ea8-config-volume\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:27.998412 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.998217 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:27.998412 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.998242 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpn99\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-kube-api-access-lpn99\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:27.998412 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:27.998286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-trusted-ca\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.037734 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.037681 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:28.099696 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.099617 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-image-registry-private-configuration\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.099696 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.099657 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:28.099696 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.099676 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-certificates\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.099973 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.099737 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-installation-pull-secrets\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.099973 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.099776 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4253f459-2c3c-427f-b9da-3449dbd7212f-ca-trust-extracted\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.099973 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.099798 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 14:48:28.099973 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.099882 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls podName:3e334195-bba1-46db-8e7b-b746dbb85ea8 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:28.59985924 +0000 UTC m=+35.583142572 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls") pod "dns-default-gx9sl" (UID: "3e334195-bba1-46db-8e7b-b746dbb85ea8") : secret "dns-default-metrics-tls" not found Jul 30 14:48:28.099973 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.099905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e334195-bba1-46db-8e7b-b746dbb85ea8-tmp-dir\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:28.100226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.099979 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7fft\" (UniqueName: \"kubernetes.io/projected/3e334195-bba1-46db-8e7b-b746dbb85ea8-kube-api-access-d7fft\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:28.100226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a-cert\") pod \"ingress-canary-2kjxw\" (UID: \"99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a\") " pod="openshift-ingress-canary/ingress-canary-2kjxw" Jul 30 14:48:28.100226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-bound-sa-token\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.100226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100103 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e334195-bba1-46db-8e7b-b746dbb85ea8-config-volume\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:28.100226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100132 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.100226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpn99\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-kube-api-access-lpn99\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.100226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100201 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-trusted-ca\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.100226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100226 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnlgj\" (UniqueName: \"kubernetes.io/projected/99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a-kube-api-access-wnlgj\") pod \"ingress-canary-2kjxw\" (UID: \"99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a\") " pod="openshift-ingress-canary/ingress-canary-2kjxw" Jul 30 14:48:28.100589 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100318 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e334195-bba1-46db-8e7b-b746dbb85ea8-tmp-dir\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:28.100589 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-certificates\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.100589 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.100553 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 14:48:28.100589 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.100568 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757dcf8f9-8j2qt: secret "image-registry-tls" not found Jul 30 14:48:28.100770 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.100624 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls podName:4253f459-2c3c-427f-b9da-3449dbd7212f nodeName:}" failed. No retries permitted until 2026-07-30 14:48:28.600605646 +0000 UTC m=+35.583888977 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls") pod "image-registry-757dcf8f9-8j2qt" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f") : secret "image-registry-tls" not found Jul 30 14:48:28.100830 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/3e334195-bba1-46db-8e7b-b746dbb85ea8-config-volume\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:28.100929 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.100908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4253f459-2c3c-427f-b9da-3449dbd7212f-ca-trust-extracted\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.101460 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.101438 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-trusted-ca\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.104927 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.104901 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-installation-pull-secrets\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.104927 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.104917 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-image-registry-private-configuration\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.105115 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.104911 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a-cert\") pod \"ingress-canary-2kjxw\" (UID: \"99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a\") " pod="openshift-ingress-canary/ingress-canary-2kjxw" Jul 30 14:48:28.108648 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.108573 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpn99\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-kube-api-access-lpn99\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.108648 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.108582 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7fft\" (UniqueName: \"kubernetes.io/projected/3e334195-bba1-46db-8e7b-b746dbb85ea8-kube-api-access-d7fft\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:28.108895 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.108876 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-bound-sa-token\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.109227 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.109209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnlgj\" (UniqueName: \"kubernetes.io/projected/99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a-kube-api-access-wnlgj\") pod \"ingress-canary-2kjxw\" (UID: \"99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a\") " pod="openshift-ingress-canary/ingress-canary-2kjxw" Jul 30 14:48:28.125499 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.125452 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:28.216538 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.216498 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2kjxw" Jul 30 14:48:28.484706 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.484664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:28.484891 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.484664 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:28.487034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.486999 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 14:48:28.487034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.487022 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dd6sm\"" Jul 30 14:48:28.487757 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.487116 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 14:48:28.487757 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.487174 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 14:48:28.604482 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.604431 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:28.604660 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.604542 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:28.604660 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.604595 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 14:48:28.604772 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.604667 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 14:48:28.604772 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.604681 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757dcf8f9-8j2qt: secret "image-registry-tls" not found Jul 30 14:48:28.604772 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.604668 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls podName:3e334195-bba1-46db-8e7b-b746dbb85ea8 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:29.604650785 +0000 UTC m=+36.587934115 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls") pod "dns-default-gx9sl" (UID: "3e334195-bba1-46db-8e7b-b746dbb85ea8") : secret "dns-default-metrics-tls" not found Jul 30 14:48:28.604772 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:28.604730 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls podName:4253f459-2c3c-427f-b9da-3449dbd7212f nodeName:}" failed. No retries permitted until 2026-07-30 14:48:29.604714434 +0000 UTC m=+36.587997748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls") pod "image-registry-757dcf8f9-8j2qt" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f") : secret "image-registry-tls" not found Jul 30 14:48:28.865442 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:28.865237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2kjxw"] Jul 30 14:48:28.869359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:48:28.869337 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e1e1d4_1ab0_47fb_a4cd_a2d99d49644a.slice/crio-df5d40d78c45bbd0ba480b4e72b9484d2e840cfa9e362ca4f16e9893dd7d7bf0 WatchSource:0}: Error finding container df5d40d78c45bbd0ba480b4e72b9484d2e840cfa9e362ca4f16e9893dd7d7bf0: Status 404 returned error can't find the container with id df5d40d78c45bbd0ba480b4e72b9484d2e840cfa9e362ca4f16e9893dd7d7bf0 Jul 30 14:48:29.484885 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:29.484849 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:29.487257 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:29.487232 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82hs8\"" Jul 30 14:48:29.487745 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:29.487267 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 14:48:29.613504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:29.613470 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:29.613705 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:29.613538 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:29.613705 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:29.613660 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 14:48:29.613705 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:29.613683 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757dcf8f9-8j2qt: secret "image-registry-tls" not found Jul 30 14:48:29.613857 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:29.613739 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls podName:4253f459-2c3c-427f-b9da-3449dbd7212f nodeName:}" failed. No retries permitted until 2026-07-30 14:48:31.613719042 +0000 UTC m=+38.597002370 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls") pod "image-registry-757dcf8f9-8j2qt" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f") : secret "image-registry-tls" not found Jul 30 14:48:29.613857 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:29.613657 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 14:48:29.613857 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:29.613825 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls podName:3e334195-bba1-46db-8e7b-b746dbb85ea8 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:31.613794326 +0000 UTC m=+38.597077659 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls") pod "dns-default-gx9sl" (UID: "3e334195-bba1-46db-8e7b-b746dbb85ea8") : secret "dns-default-metrics-tls" not found Jul 30 14:48:29.723595 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:29.723561 2564 generic.go:356] "Generic (PLEG): container finished" podID="22a81aec-43f6-4bba-8661-5b4785b398bc" containerID="e82dcb9a14182e793465bddf11ca2ee281c9145f2443f1a559582f555fbc509e" exitCode=0 Jul 30 14:48:29.723809 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:29.723659 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" event={"ID":"22a81aec-43f6-4bba-8661-5b4785b398bc","Type":"ContainerDied","Data":"e82dcb9a14182e793465bddf11ca2ee281c9145f2443f1a559582f555fbc509e"} Jul 30 14:48:29.729062 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:29.729002 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2kjxw" event={"ID":"99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a","Type":"ContainerStarted","Data":"df5d40d78c45bbd0ba480b4e72b9484d2e840cfa9e362ca4f16e9893dd7d7bf0"} Jul 30 14:48:30.736641 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:30.736610 2564 generic.go:356] "Generic (PLEG): container finished" podID="22a81aec-43f6-4bba-8661-5b4785b398bc" containerID="c3a0e02b50f9eb7b5653a26e9815e25f005246d3155ca6f9b5f8c036c1bde550" exitCode=0 Jul 30 14:48:30.737050 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:30.736657 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" event={"ID":"22a81aec-43f6-4bba-8661-5b4785b398bc","Type":"ContainerDied","Data":"c3a0e02b50f9eb7b5653a26e9815e25f005246d3155ca6f9b5f8c036c1bde550"} Jul 30 14:48:31.275899 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:31.275820 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:31.334119 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:31.334068 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:31.630152 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:31.630119 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:31.630315 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:31.630171 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:31.630315 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:31.630292 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 14:48:31.630447 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:31.630318 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757dcf8f9-8j2qt: secret "image-registry-tls" not found Jul 30 14:48:31.630447 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:31.630365 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 14:48:31.630447 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:31.630390 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls podName:4253f459-2c3c-427f-b9da-3449dbd7212f nodeName:}" failed. No retries permitted until 2026-07-30 14:48:35.630368447 +0000 UTC m=+42.613651782 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls") pod "image-registry-757dcf8f9-8j2qt" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f") : secret "image-registry-tls" not found Jul 30 14:48:31.630447 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:31.630408 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls podName:3e334195-bba1-46db-8e7b-b746dbb85ea8 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:35.630396951 +0000 UTC m=+42.613680265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls") pod "dns-default-gx9sl" (UID: "3e334195-bba1-46db-8e7b-b746dbb85ea8") : secret "dns-default-metrics-tls" not found Jul 30 14:48:31.744295 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:31.744042 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" event={"ID":"22a81aec-43f6-4bba-8661-5b4785b398bc","Type":"ContainerStarted","Data":"fa9aad0ff978e5f86e49417f83bc409240cab2a3769f2ef950e8734b0ade9eeb"} Jul 30 14:48:31.745614 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:31.745589 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2kjxw" event={"ID":"99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a","Type":"ContainerStarted","Data":"7e5d2da8b8e3b31b21de647d096ddce2d165c276d47cf63c405dcbdce4072839"} Jul 30 14:48:31.778191 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:31.778137 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-bm8gg" podStartSLOduration=6.265519565 podStartE2EDuration="37.778119808s" totalImagesPullingTime="31.512600243s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:31.777184343 +0000 UTC m=+38.760467681" watchObservedRunningTime="2026-07-30 14:48:31.778119808 +0000 UTC m=+38.761403146" Jul 30 14:48:31.800563 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:31.800460 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2kjxw" podStartSLOduration=2.153942734 podStartE2EDuration="4.80044463s" totalImagesPullingTime="2.646501896s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:48:31.799197966 +0000 UTC m=+38.782481314" watchObservedRunningTime="2026-07-30 14:48:31.80044463 +0000 UTC m=+38.783728014" Jul 30 14:48:32.023817 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:32.023781 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:32.104402 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:32.104317 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:32.232282 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:32.232244 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:32.632477 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:32.632426 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:32.728866 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:32.728833 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:33.141758 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:33.141719 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:35.660833 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:35.660791 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:35.661373 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:35.660870 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:35.661373 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:35.660958 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 14:48:35.661373 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:35.660969 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757dcf8f9-8j2qt: secret "image-registry-tls" not found Jul 30 14:48:35.661373 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:35.660967 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 14:48:35.661373 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:35.661018 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls podName:4253f459-2c3c-427f-b9da-3449dbd7212f nodeName:}" failed. No retries permitted until 2026-07-30 14:48:43.661003873 +0000 UTC m=+50.644287189 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls") pod "image-registry-757dcf8f9-8j2qt" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f") : secret "image-registry-tls" not found Jul 30 14:48:35.661373 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:35.661030 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls podName:3e334195-bba1-46db-8e7b-b746dbb85ea8 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:43.661024236 +0000 UTC m=+50.644307550 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls") pod "dns-default-gx9sl" (UID: "3e334195-bba1-46db-8e7b-b746dbb85ea8") : secret "dns-default-metrics-tls" not found Jul 30 14:48:36.001427 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:36.001391 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:36.102024 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:36.101991 2564 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" limit=1024 Jul 30 14:48:43.722397 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:43.722294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:43.722871 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:43.722403 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:43.722871 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:43.722457 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 14:48:43.722871 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:43.722523 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls podName:3e334195-bba1-46db-8e7b-b746dbb85ea8 nodeName:}" failed. No retries permitted until 2026-07-30 14:48:59.722505581 +0000 UTC m=+66.705788895 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls") pod "dns-default-gx9sl" (UID: "3e334195-bba1-46db-8e7b-b746dbb85ea8") : secret "dns-default-metrics-tls" not found Jul 30 14:48:43.722871 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:43.722537 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 14:48:43.722871 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:43.722556 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757dcf8f9-8j2qt: secret "image-registry-tls" not found Jul 30 14:48:43.722871 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:43.722616 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls podName:4253f459-2c3c-427f-b9da-3449dbd7212f nodeName:}" failed. No retries permitted until 2026-07-30 14:48:59.722597389 +0000 UTC m=+66.705880721 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls") pod "image-registry-757dcf8f9-8j2qt" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f") : secret "image-registry-tls" not found Jul 30 14:48:53.723442 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:53.723416 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-6r57d" Jul 30 14:48:58.230363 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.230301 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:58.232675 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.232657 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 30 14:48:58.244743 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.244707 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0048d797-3821-4f51-92ed-84cb6b974158-original-pull-secret\") pod \"global-pull-secret-syncer-p86cc\" (UID: \"0048d797-3821-4f51-92ed-84cb6b974158\") " pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:58.330700 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.330647 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:58.330914 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.330736 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:48:58.333418 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.333394 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 30 14:48:58.333418 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.333411 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 30 14:48:58.341130 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:58.341106 2564 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 30 14:48:58.341198 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:58.341183 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs podName:bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71 nodeName:}" failed. No retries permitted until 2026-07-30 14:50:02.341165897 +0000 UTC m=+129.324449215 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs") pod "network-metrics-daemon-4k86d" (UID: "bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71") : secret "metrics-daemon-secret" not found Jul 30 14:48:58.342842 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.342828 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 30 14:48:58.353843 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.353813 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-scpv7\" (UniqueName: \"kubernetes.io/projected/e4815781-a404-4160-b0e6-dd4f866d0ca4-kube-api-access-scpv7\") pod \"network-check-target-gm8xd\" (UID: \"e4815781-a404-4160-b0e6-dd4f866d0ca4\") " pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:58.499442 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.499364 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-dd6sm\"" Jul 30 14:48:58.502822 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.502798 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p86cc" Jul 30 14:48:58.507728 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.507703 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:48:58.659482 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.659442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p86cc"] Jul 30 14:48:58.663080 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:48:58.663044 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0048d797_3821_4f51_92ed_84cb6b974158.slice/crio-d38decd993f1fc5d5075d4fa6642f81aa0072669b414acb410d3c2a41bae9bab WatchSource:0}: Error finding container d38decd993f1fc5d5075d4fa6642f81aa0072669b414acb410d3c2a41bae9bab: Status 404 returned error can't find the container with id d38decd993f1fc5d5075d4fa6642f81aa0072669b414acb410d3c2a41bae9bab Jul 30 14:48:58.671434 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.671413 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gm8xd"] Jul 30 14:48:58.675274 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:48:58.675250 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4815781_a404_4160_b0e6_dd4f866d0ca4.slice/crio-1700b4c111019810816698104a1ba74808da4d2e4a5a300bff760d4b82558ffe WatchSource:0}: Error finding container 1700b4c111019810816698104a1ba74808da4d2e4a5a300bff760d4b82558ffe: Status 404 returned error can't find the container with id 1700b4c111019810816698104a1ba74808da4d2e4a5a300bff760d4b82558ffe Jul 30 14:48:58.811406 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.811293 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gm8xd" event={"ID":"e4815781-a404-4160-b0e6-dd4f866d0ca4","Type":"ContainerStarted","Data":"1700b4c111019810816698104a1ba74808da4d2e4a5a300bff760d4b82558ffe"} Jul 30 14:48:58.812342 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:58.812305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p86cc" event={"ID":"0048d797-3821-4f51-92ed-84cb6b974158","Type":"ContainerStarted","Data":"d38decd993f1fc5d5075d4fa6642f81aa0072669b414acb410d3c2a41bae9bab"} Jul 30 14:48:59.740282 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:59.740246 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:48:59.740736 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:48:59.740299 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:48:59.740736 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:59.740432 2564 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 30 14:48:59.740736 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:59.740506 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls podName:3e334195-bba1-46db-8e7b-b746dbb85ea8 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:31.740486991 +0000 UTC m=+98.723770326 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls") pod "dns-default-gx9sl" (UID: "3e334195-bba1-46db-8e7b-b746dbb85ea8") : secret "dns-default-metrics-tls" not found Jul 30 14:48:59.740736 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:59.740435 2564 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 30 14:48:59.740736 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:59.740550 2564 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-757dcf8f9-8j2qt: secret "image-registry-tls" not found Jul 30 14:48:59.740736 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:48:59.740612 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls podName:4253f459-2c3c-427f-b9da-3449dbd7212f nodeName:}" failed. No retries permitted until 2026-07-30 14:49:31.740591548 +0000 UTC m=+98.723874879 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls") pod "image-registry-757dcf8f9-8j2qt" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f") : secret "image-registry-tls" not found Jul 30 14:49:01.058717 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.058398 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52"] Jul 30 14:49:01.084545 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.084483 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52"] Jul 30 14:49:01.084722 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.084630 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.087220 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.087194 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-xbtft\"" Jul 30 14:49:01.087369 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.087266 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 30 14:49:01.087807 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.087784 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 30 14:49:01.088041 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.088019 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 30 14:49:01.088388 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.088222 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 30 14:49:01.153382 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.153339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ade968f1-c7b5-45c7-b939-a7b2213bd06b-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.153572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.153452 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7d9n\" (UniqueName: \"kubernetes.io/projected/ade968f1-c7b5-45c7-b939-a7b2213bd06b-kube-api-access-w7d9n\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.153572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.153494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade968f1-c7b5-45c7-b939-a7b2213bd06b-config\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.157579 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.157547 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d"] Jul 30 14:49:01.176424 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.176344 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh"] Jul 30 14:49:01.176974 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.176944 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.180254 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.179971 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 30 14:49:01.180254 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.180020 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 30 14:49:01.180254 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.179972 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 30 14:49:01.180479 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.180286 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 30 14:49:01.180479 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.180332 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-npnc8\"" Jul 30 14:49:01.194429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.194399 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh"] Jul 30 14:49:01.194742 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.194445 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d"] Jul 30 14:49:01.194742 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.194559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:01.196508 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.196347 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-bnnz7\"" Jul 30 14:49:01.196883 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.196713 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 30 14:49:01.196883 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.196759 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 30 14:49:01.196883 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.196798 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 30 14:49:01.253924 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.253890 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ade968f1-c7b5-45c7-b939-a7b2213bd06b-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.253924 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.253926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msjkm\" (UniqueName: \"kubernetes.io/projected/985d2b4d-60af-46d8-9f00-6d8552acee68-kube-api-access-msjkm\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:01.254193 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.253989 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7d9n\" (UniqueName: \"kubernetes.io/projected/ade968f1-c7b5-45c7-b939-a7b2213bd06b-kube-api-access-w7d9n\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.254193 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.254008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade968f1-c7b5-45c7-b939-a7b2213bd06b-config\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.254193 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.254028 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:01.254193 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.254095 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrsw\" (UniqueName: \"kubernetes.io/projected/31303ff1-ec5b-491b-8810-ea15f989b7c8-kube-api-access-kdrsw\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.254193 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.254129 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.254193 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.254154 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/31303ff1-ec5b-491b-8810-ea15f989b7c8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.254963 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.254934 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ade968f1-c7b5-45c7-b939-a7b2213bd06b-config\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.256880 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.256852 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ade968f1-c7b5-45c7-b939-a7b2213bd06b-serving-cert\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.262845 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.262795 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7d9n\" (UniqueName: \"kubernetes.io/projected/ade968f1-c7b5-45c7-b939-a7b2213bd06b-kube-api-access-w7d9n\") pod \"service-ca-operator-5d6c75f84f-vkj52\" (UID: \"ade968f1-c7b5-45c7-b939-a7b2213bd06b\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.354903 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.354811 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrsw\" (UniqueName: \"kubernetes.io/projected/31303ff1-ec5b-491b-8810-ea15f989b7c8-kube-api-access-kdrsw\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.354903 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.354851 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.354903 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.354881 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/31303ff1-ec5b-491b-8810-ea15f989b7c8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.355279 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.354947 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msjkm\" (UniqueName: \"kubernetes.io/projected/985d2b4d-60af-46d8-9f00-6d8552acee68-kube-api-access-msjkm\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:01.355279 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:01.354983 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:01.355279 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.355008 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:01.355279 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:01.355058 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls podName:31303ff1-ec5b-491b-8810-ea15f989b7c8 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:01.855037729 +0000 UTC m=+68.838321048 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2qq5d" (UID: "31303ff1-ec5b-491b-8810-ea15f989b7c8") : secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:01.355279 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:01.355123 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 14:49:01.355279 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:01.355195 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls podName:985d2b4d-60af-46d8-9f00-6d8552acee68 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:01.855179839 +0000 UTC m=+68.838463175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls") pod "cluster-samples-operator-6748489656-sk2jh" (UID: "985d2b4d-60af-46d8-9f00-6d8552acee68") : secret "samples-operator-tls" not found Jul 30 14:49:01.356001 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.355757 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/31303ff1-ec5b-491b-8810-ea15f989b7c8-telemetry-config\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.364348 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.364324 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msjkm\" (UniqueName: \"kubernetes.io/projected/985d2b4d-60af-46d8-9f00-6d8552acee68-kube-api-access-msjkm\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:01.364476 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.364427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrsw\" (UniqueName: \"kubernetes.io/projected/31303ff1-ec5b-491b-8810-ea15f989b7c8-kube-api-access-kdrsw\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.397269 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.397235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" Jul 30 14:49:01.550783 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.550747 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52"] Jul 30 14:49:01.858730 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.858694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:01.858897 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:01.858768 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:01.858962 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:01.858885 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 14:49:01.858962 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:01.858917 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:01.859039 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:01.858972 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls podName:985d2b4d-60af-46d8-9f00-6d8552acee68 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:02.858949878 +0000 UTC m=+69.842233196 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls") pod "cluster-samples-operator-6748489656-sk2jh" (UID: "985d2b4d-60af-46d8-9f00-6d8552acee68") : secret "samples-operator-tls" not found Jul 30 14:49:01.859039 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:01.858992 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls podName:31303ff1-ec5b-491b-8810-ea15f989b7c8 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:02.8589825 +0000 UTC m=+69.842265819 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2qq5d" (UID: "31303ff1-ec5b-491b-8810-ea15f989b7c8") : secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:02.866215 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:02.866165 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:02.866704 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:02.866337 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 14:49:02.866704 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:02.866348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:02.866704 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:02.866485 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls podName:985d2b4d-60af-46d8-9f00-6d8552acee68 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:04.866465153 +0000 UTC m=+71.849748472 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls") pod "cluster-samples-operator-6748489656-sk2jh" (UID: "985d2b4d-60af-46d8-9f00-6d8552acee68") : secret "samples-operator-tls" not found Jul 30 14:49:02.866704 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:02.866541 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:02.866704 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:02.866603 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls podName:31303ff1-ec5b-491b-8810-ea15f989b7c8 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:04.866589834 +0000 UTC m=+71.849873149 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2qq5d" (UID: "31303ff1-ec5b-491b-8810-ea15f989b7c8") : secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:02.960534 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:02.960501 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podade968f1_c7b5_45c7_b939_a7b2213bd06b.slice/crio-dd296793a3ce41fdf9b06ef70e430af2693f273e2a92d7e0f4551a8a60a6a578 WatchSource:0}: Error finding container dd296793a3ce41fdf9b06ef70e430af2693f273e2a92d7e0f4551a8a60a6a578: Status 404 returned error can't find the container with id dd296793a3ce41fdf9b06ef70e430af2693f273e2a92d7e0f4551a8a60a6a578 Jul 30 14:49:03.826000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:03.825955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" event={"ID":"ade968f1-c7b5-45c7-b939-a7b2213bd06b","Type":"ContainerStarted","Data":"dd296793a3ce41fdf9b06ef70e430af2693f273e2a92d7e0f4551a8a60a6a578"} Jul 30 14:49:04.829540 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:04.829503 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p86cc" event={"ID":"0048d797-3821-4f51-92ed-84cb6b974158","Type":"ContainerStarted","Data":"c9793e5c499b4eee1ff244a20b3ba8571a994cddb545e1c34eb31162cebd2799"} Jul 30 14:49:04.830959 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:04.830935 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gm8xd" event={"ID":"e4815781-a404-4160-b0e6-dd4f866d0ca4","Type":"ContainerStarted","Data":"10da137b8c2bda4d6ffbf09962d96bf40a624f04cf61c9b100f4504fb2a29801"} Jul 30 14:49:04.831084 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:04.831059 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:49:04.842803 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:04.842759 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p86cc" podStartSLOduration=65.548234907 podStartE2EDuration="1m10.842744223s" totalImagesPullingTime="5.294509316s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:04.841894863 +0000 UTC m=+71.825178199" watchObservedRunningTime="2026-07-30 14:49:04.842744223 +0000 UTC m=+71.826027554" Jul 30 14:49:04.855662 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:04.855619 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gm8xd" podStartSLOduration=65.574833691 podStartE2EDuration="1m10.855605447s" totalImagesPullingTime="5.280771756s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:04.855008689 +0000 UTC m=+71.838292026" watchObservedRunningTime="2026-07-30 14:49:04.855605447 +0000 UTC m=+71.838888804" Jul 30 14:49:04.885000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:04.884961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:04.885172 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:04.885037 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:04.885305 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:04.885207 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:04.885305 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:04.885280 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls podName:31303ff1-ec5b-491b-8810-ea15f989b7c8 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:08.885259136 +0000 UTC m=+75.868542475 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2qq5d" (UID: "31303ff1-ec5b-491b-8810-ea15f989b7c8") : secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:04.885419 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:04.885322 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 14:49:04.885419 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:04.885393 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls podName:985d2b4d-60af-46d8-9f00-6d8552acee68 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:08.88537232 +0000 UTC m=+75.868655650 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls") pod "cluster-samples-operator-6748489656-sk2jh" (UID: "985d2b4d-60af-46d8-9f00-6d8552acee68") : secret "samples-operator-tls" not found Jul 30 14:49:05.835488 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:05.835451 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" event={"ID":"ade968f1-c7b5-45c7-b939-a7b2213bd06b","Type":"ContainerStarted","Data":"b5080ef956ad89f159c9505f9abfe2e3e8f581483ac8df0515c2913598aa80fb"} Jul 30 14:49:05.851111 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:05.851043 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" podStartSLOduration=2.200500392 podStartE2EDuration="4.851027666s" totalImagesPullingTime="2.650527274s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:05.84989533 +0000 UTC m=+72.833178668" watchObservedRunningTime="2026-07-30 14:49:05.851027666 +0000 UTC m=+72.834311002" Jul 30 14:49:06.104053 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.104022 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m"] Jul 30 14:49:06.105918 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.105901 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" Jul 30 14:49:06.108627 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.108597 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 30 14:49:06.108748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.108669 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 30 14:49:06.108748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.108676 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-9sg28\"" Jul 30 14:49:06.116463 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.116442 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m"] Jul 30 14:49:06.181792 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.181758 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-f8s2b"] Jul 30 14:49:06.185171 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.185145 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.187261 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.187190 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 30 14:49:06.187261 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.187208 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 30 14:49:06.187470 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.187261 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 30 14:49:06.187526 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.187475 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9mfqr\"" Jul 30 14:49:06.187526 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.187481 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 30 14:49:06.196462 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.196440 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8s2b"] Jul 30 14:49:06.196710 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.196693 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btq62\" (UniqueName: \"kubernetes.io/projected/20224d9f-7519-4467-aacf-3a6d0675d45b-kube-api-access-btq62\") pod \"migrator-6865c9967-zpt7m\" (UID: \"20224d9f-7519-4467-aacf-3a6d0675d45b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" Jul 30 14:49:06.297213 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.297178 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btq62\" (UniqueName: \"kubernetes.io/projected/20224d9f-7519-4467-aacf-3a6d0675d45b-kube-api-access-btq62\") pod \"migrator-6865c9967-zpt7m\" (UID: \"20224d9f-7519-4467-aacf-3a6d0675d45b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" Jul 30 14:49:06.297401 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.297235 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwph7\" (UniqueName: \"kubernetes.io/projected/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-api-access-hwph7\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.297401 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.297259 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a92369c-66e2-4c03-a22d-8ef9cc74231c-data-volume\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.297482 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.297408 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.297521 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.297497 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a92369c-66e2-4c03-a22d-8ef9cc74231c-crio-socket\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.297554 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.297528 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.304681 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.304657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-btq62\" (UniqueName: \"kubernetes.io/projected/20224d9f-7519-4467-aacf-3a6d0675d45b-kube-api-access-btq62\") pod \"migrator-6865c9967-zpt7m\" (UID: \"20224d9f-7519-4467-aacf-3a6d0675d45b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" Jul 30 14:49:06.398475 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.398386 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwph7\" (UniqueName: \"kubernetes.io/projected/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-api-access-hwph7\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.398475 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.398425 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a92369c-66e2-4c03-a22d-8ef9cc74231c-data-volume\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.398475 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.398452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.398734 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.398508 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a92369c-66e2-4c03-a22d-8ef9cc74231c-crio-socket\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.398734 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.398525 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.398734 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:06.398610 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 14:49:06.398734 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.398654 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3a92369c-66e2-4c03-a22d-8ef9cc74231c-crio-socket\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.398734 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:06.398676 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:06.898655545 +0000 UTC m=+73.881938880 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : configmap "kube-rbac-proxy" not found Jul 30 14:49:06.398734 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:06.398706 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 14:49:06.398962 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:06.398751 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:06.898739104 +0000 UTC m=+73.882022438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : secret "insights-runtime-extractor-tls" not found Jul 30 14:49:06.398962 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.398772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3a92369c-66e2-4c03-a22d-8ef9cc74231c-data-volume\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.408693 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.408665 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwph7\" (UniqueName: \"kubernetes.io/projected/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-api-access-hwph7\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.414989 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.414969 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" Jul 30 14:49:06.542708 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.542677 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m"] Jul 30 14:49:06.546177 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:06.546151 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20224d9f_7519_4467_aacf_3a6d0675d45b.slice/crio-c22836d8d70ec2cc1d88d88f635538122531bc11cee26ea2030571f866a11077 WatchSource:0}: Error finding container c22836d8d70ec2cc1d88d88f635538122531bc11cee26ea2030571f866a11077: Status 404 returned error can't find the container with id c22836d8d70ec2cc1d88d88f635538122531bc11cee26ea2030571f866a11077 Jul 30 14:49:06.838627 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.838593 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" event={"ID":"20224d9f-7519-4467-aacf-3a6d0675d45b","Type":"ContainerStarted","Data":"c22836d8d70ec2cc1d88d88f635538122531bc11cee26ea2030571f866a11077"} Jul 30 14:49:06.903950 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.903908 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.904147 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:06.903996 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 14:49:06.904147 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:06.904004 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:06.904147 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:06.904086 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:07.90404832 +0000 UTC m=+74.887331652 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : configmap "kube-rbac-proxy" not found Jul 30 14:49:06.904298 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:06.904266 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 14:49:06.904338 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:06.904312 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:07.904298706 +0000 UTC m=+74.887582021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : secret "insights-runtime-extractor-tls" not found Jul 30 14:49:07.911298 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:07.911267 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:07.911619 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:07.911326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:07.911619 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:07.911393 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 14:49:07.911619 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:07.911434 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 14:49:07.911619 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:07.911466 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:09.911445564 +0000 UTC m=+76.894728881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : configmap "kube-rbac-proxy" not found Jul 30 14:49:07.911619 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:07.911482 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:09.9114757 +0000 UTC m=+76.894759015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : secret "insights-runtime-extractor-tls" not found Jul 30 14:49:08.845326 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:08.845295 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" event={"ID":"20224d9f-7519-4467-aacf-3a6d0675d45b","Type":"ContainerStarted","Data":"44e75d3aa2cd62fec5d9b6cf1e0756f98314004c8c15ee4f487a36b763e425d2"} Jul 30 14:49:08.845502 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:08.845352 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" event={"ID":"20224d9f-7519-4467-aacf-3a6d0675d45b","Type":"ContainerStarted","Data":"dcda815218460aedcc3c24e2f45b965038ba8341ffe34518c41beaee19dc9278"} Jul 30 14:49:08.861995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:08.861950 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-zpt7m" podStartSLOduration=1.620020694 podStartE2EDuration="2.861934425s" totalImagesPullingTime="1.241913731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:08.8614843 +0000 UTC m=+75.844767638" watchObservedRunningTime="2026-07-30 14:49:08.861934425 +0000 UTC m=+75.845217762" Jul 30 14:49:08.921360 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:08.921328 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:08.921708 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:08.921368 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:08.921708 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:08.921497 2564 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 30 14:49:08.921708 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:08.921530 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:08.921708 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:08.921575 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls podName:985d2b4d-60af-46d8-9f00-6d8552acee68 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:16.921555136 +0000 UTC m=+83.904838466 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls") pod "cluster-samples-operator-6748489656-sk2jh" (UID: "985d2b4d-60af-46d8-9f00-6d8552acee68") : secret "samples-operator-tls" not found Jul 30 14:49:08.921708 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:08.921589 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls podName:31303ff1-ec5b-491b-8810-ea15f989b7c8 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:16.921583179 +0000 UTC m=+83.904866493 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2qq5d" (UID: "31303ff1-ec5b-491b-8810-ea15f989b7c8") : secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:08.989499 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:08.989471 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dhkrf_36159941-b1b2-4d47-98ce-cf212bf196a2/dns-node-resolver/0.log" Jul 30 14:49:09.930192 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:09.930145 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:09.930585 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:09.930214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:09.930585 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:09.930294 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 14:49:09.930585 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:09.930313 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 14:49:09.930585 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:09.930359 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:13.930343382 +0000 UTC m=+80.913626700 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : configmap "kube-rbac-proxy" not found Jul 30 14:49:09.930585 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:09.930433 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:13.930407713 +0000 UTC m=+80.913691032 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : secret "insights-runtime-extractor-tls" not found Jul 30 14:49:09.989605 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:09.989578 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxskg_771dcee2-efa1-4503-9e75-16b904ed3df2/node-ca/0.log" Jul 30 14:49:10.588999 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:10.588968 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2kjxw_99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a/serve-healthcheck-canary/0.log" Jul 30 14:49:11.191575 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:11.191544 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zpt7m_20224d9f-7519-4467-aacf-3a6d0675d45b/migrator/0.log" Jul 30 14:49:11.388991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:11.388963 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zpt7m_20224d9f-7519-4467-aacf-3a6d0675d45b/graceful-termination/0.log" Jul 30 14:49:13.961678 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:13.961638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:13.962099 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:13.961725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:13.962099 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:13.961794 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 14:49:13.962099 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:13.961871 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:21.961854344 +0000 UTC m=+88.945137662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : secret "insights-runtime-extractor-tls" not found Jul 30 14:49:13.962099 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:13.961800 2564 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 30 14:49:13.962099 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:13.961930 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:21.961918068 +0000 UTC m=+88.945201410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : configmap "kube-rbac-proxy" not found Jul 30 14:49:16.986595 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:16.986464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:16.986595 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:16.986519 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:16.987006 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:16.986618 2564 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:16.987006 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:16.986677 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls podName:31303ff1-ec5b-491b-8810-ea15f989b7c8 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:32.986664049 +0000 UTC m=+99.969947377 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-548b489cb8-2qq5d" (UID: "31303ff1-ec5b-491b-8810-ea15f989b7c8") : secret "cluster-monitoring-operator-tls" not found Jul 30 14:49:16.988870 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:16.988848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/985d2b4d-60af-46d8-9f00-6d8552acee68-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-sk2jh\" (UID: \"985d2b4d-60af-46d8-9f00-6d8552acee68\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:17.107439 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:17.107405 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" Jul 30 14:49:17.227441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:17.227407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh"] Jul 30 14:49:17.876003 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:17.875965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" event={"ID":"985d2b4d-60af-46d8-9f00-6d8552acee68","Type":"ContainerStarted","Data":"e9c4783717683a35bf0e60aebc14600f6c3ecc8da73a113fece1aebaf05a7895"} Jul 30 14:49:20.888343 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:20.888304 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" event={"ID":"985d2b4d-60af-46d8-9f00-6d8552acee68","Type":"ContainerStarted","Data":"cfc99e2b74eefb092b07142267ea6624e43a2dc3605e961bb9d8e6b8e5f601ea"} Jul 30 14:49:20.888824 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:20.888390 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" event={"ID":"985d2b4d-60af-46d8-9f00-6d8552acee68","Type":"ContainerStarted","Data":"c1a06ac5ce6bc66f48f1ca155f2f2807cee8314ec232785197e12e1e34b57b53"} Jul 30 14:49:20.905126 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:20.905054 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-sk2jh" podStartSLOduration=17.325600827 podStartE2EDuration="19.905038104s" totalImagesPullingTime="2.579437277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:20.903778026 +0000 UTC m=+87.887061363" watchObservedRunningTime="2026-07-30 14:49:20.905038104 +0000 UTC m=+87.888321441" Jul 30 14:49:22.028271 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:22.028228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:22.028729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:22.028327 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:22.028729 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:22.028405 2564 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 30 14:49:22.028729 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:22.028476 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls podName:3a92369c-66e2-4c03-a22d-8ef9cc74231c nodeName:}" failed. No retries permitted until 2026-07-30 14:49:38.028459844 +0000 UTC m=+105.011743162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls") pod "insights-runtime-extractor-f8s2b" (UID: "3a92369c-66e2-4c03-a22d-8ef9cc74231c") : secret "insights-runtime-extractor-tls" not found Jul 30 14:49:22.028841 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:22.028798 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3a92369c-66e2-4c03-a22d-8ef9cc74231c-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:31.804819 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:31.804775 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:49:31.804819 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:31.804837 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:49:31.807162 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:31.807134 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/3e334195-bba1-46db-8e7b-b746dbb85ea8-metrics-tls\") pod \"dns-default-gx9sl\" (UID: \"3e334195-bba1-46db-8e7b-b746dbb85ea8\") " pod="openshift-dns/dns-default-gx9sl" Jul 30 14:49:31.807273 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:31.807202 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"image-registry-757dcf8f9-8j2qt\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:49:31.815201 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:31.815181 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-vrbw2\"" Jul 30 14:49:31.821053 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:31.821038 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-gx9sl" Jul 30 14:49:31.945316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:31.945230 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-gx9sl"] Jul 30 14:49:31.949779 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:31.949748 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e334195_bba1_46db_8e7b_b746dbb85ea8.slice/crio-ea210149d8f7311b751e424d784c8d059d373b28ff25ca69b5519a40680d5a78 WatchSource:0}: Error finding container ea210149d8f7311b751e424d784c8d059d373b28ff25ca69b5519a40680d5a78: Status 404 returned error can't find the container with id ea210149d8f7311b751e424d784c8d059d373b28ff25ca69b5519a40680d5a78 Jul 30 14:49:32.100266 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:32.100238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fxhsd\"" Jul 30 14:49:32.109181 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:32.109156 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:49:32.233051 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:32.233024 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-757dcf8f9-8j2qt"] Jul 30 14:49:32.236085 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:32.236041 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4253f459_2c3c_427f_b9da_3449dbd7212f.slice/crio-d0954975813ab292586e30edfca4fd9d4c69f2aa8e81aeef800df7b16bf1eb6a WatchSource:0}: Error finding container d0954975813ab292586e30edfca4fd9d4c69f2aa8e81aeef800df7b16bf1eb6a: Status 404 returned error can't find the container with id d0954975813ab292586e30edfca4fd9d4c69f2aa8e81aeef800df7b16bf1eb6a Jul 30 14:49:32.924240 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:32.924203 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx9sl" event={"ID":"3e334195-bba1-46db-8e7b-b746dbb85ea8","Type":"ContainerStarted","Data":"ea210149d8f7311b751e424d784c8d059d373b28ff25ca69b5519a40680d5a78"} Jul 30 14:49:32.925495 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:32.925471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" event={"ID":"4253f459-2c3c-427f-b9da-3449dbd7212f","Type":"ContainerStarted","Data":"c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa"} Jul 30 14:49:32.925610 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:32.925516 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" event={"ID":"4253f459-2c3c-427f-b9da-3449dbd7212f","Type":"ContainerStarted","Data":"d0954975813ab292586e30edfca4fd9d4c69f2aa8e81aeef800df7b16bf1eb6a"} Jul 30 14:49:32.925610 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:32.925598 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:49:33.012952 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:33.012913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:33.016017 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:33.015992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/31303ff1-ec5b-491b-8810-ea15f989b7c8-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-548b489cb8-2qq5d\" (UID: \"31303ff1-ec5b-491b-8810-ea15f989b7c8\") " pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:33.288576 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:33.288543 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" Jul 30 14:49:33.421988 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:33.421938 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" podStartSLOduration=99.421915083 podStartE2EDuration="1m39.421915083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:32.946402106 +0000 UTC m=+99.929685443" watchObservedRunningTime="2026-07-30 14:49:33.421915083 +0000 UTC m=+100.405198421" Jul 30 14:49:33.422460 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:33.422438 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d"] Jul 30 14:49:33.836235 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:33.836200 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31303ff1_ec5b_491b_8810_ea15f989b7c8.slice/crio-2a80b647b33129869548feb60407cdd32ef7117451eb1cabcfa40df3ea4bb2d9 WatchSource:0}: Error finding container 2a80b647b33129869548feb60407cdd32ef7117451eb1cabcfa40df3ea4bb2d9: Status 404 returned error can't find the container with id 2a80b647b33129869548feb60407cdd32ef7117451eb1cabcfa40df3ea4bb2d9 Jul 30 14:49:33.928908 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:33.928875 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" event={"ID":"31303ff1-ec5b-491b-8810-ea15f989b7c8","Type":"ContainerStarted","Data":"2a80b647b33129869548feb60407cdd32ef7117451eb1cabcfa40df3ea4bb2d9"} Jul 30 14:49:34.934238 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:34.934199 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx9sl" event={"ID":"3e334195-bba1-46db-8e7b-b746dbb85ea8","Type":"ContainerStarted","Data":"6100e191b27141e1de3771b1b18013489813d9e50498a19a1eee18632f86d788"} Jul 30 14:49:34.934706 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:34.934265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-gx9sl" event={"ID":"3e334195-bba1-46db-8e7b-b746dbb85ea8","Type":"ContainerStarted","Data":"983c65429262452df2256639979fb55a301e5a2ef8b1ecb51d718e58c1d78b8a"} Jul 30 14:49:34.934706 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:34.934329 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-gx9sl" Jul 30 14:49:34.950155 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:34.950105 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-gx9sl" podStartSLOduration=66.020340622 podStartE2EDuration="1m7.950088873s" totalImagesPullingTime="1.929748251s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:48:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:34.948961833 +0000 UTC m=+101.932245172" watchObservedRunningTime="2026-07-30 14:49:34.950088873 +0000 UTC m=+101.933372203" Jul 30 14:49:35.838306 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:35.838278 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gm8xd" Jul 30 14:49:36.665539 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.665505 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf"] Jul 30 14:49:36.679696 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.679671 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf"] Jul 30 14:49:36.679862 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.679806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" Jul 30 14:49:36.682126 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.682100 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-9h7j4\"" Jul 30 14:49:36.682244 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.682174 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 30 14:49:36.849399 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.849365 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/514622fb-2dfc-4ac3-bad9-3e596a838b82-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-rbnbf\" (UID: \"514622fb-2dfc-4ac3-bad9-3e596a838b82\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" Jul 30 14:49:36.944417 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.944328 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" event={"ID":"31303ff1-ec5b-491b-8810-ea15f989b7c8","Type":"ContainerStarted","Data":"4081bcfb415aca1c0897db72afa2844e50180113e239e92b6abf785b6557f35d"} Jul 30 14:49:36.950559 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.950532 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/514622fb-2dfc-4ac3-bad9-3e596a838b82-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-rbnbf\" (UID: \"514622fb-2dfc-4ac3-bad9-3e596a838b82\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" Jul 30 14:49:36.953000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.952969 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/514622fb-2dfc-4ac3-bad9-3e596a838b82-tls-certificates\") pod \"prometheus-operator-admission-webhook-7f8d78f54b-rbnbf\" (UID: \"514622fb-2dfc-4ac3-bad9-3e596a838b82\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" Jul 30 14:49:36.965909 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.965857 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-548b489cb8-2qq5d" podStartSLOduration=33.663589148 podStartE2EDuration="35.965837154s" totalImagesPullingTime="2.302248006s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:36.965525046 +0000 UTC m=+103.948808384" watchObservedRunningTime="2026-07-30 14:49:36.965837154 +0000 UTC m=+103.949120491" Jul 30 14:49:36.988800 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:36.988779 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" Jul 30 14:49:37.145224 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:37.145196 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf"] Jul 30 14:49:37.148851 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:37.148818 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod514622fb_2dfc_4ac3_bad9_3e596a838b82.slice/crio-5fb5ebbd9cb3c2a74427f19c3930e9d2f13e8c039bd3cb28160b6e8d03100477 WatchSource:0}: Error finding container 5fb5ebbd9cb3c2a74427f19c3930e9d2f13e8c039bd3cb28160b6e8d03100477: Status 404 returned error can't find the container with id 5fb5ebbd9cb3c2a74427f19c3930e9d2f13e8c039bd3cb28160b6e8d03100477 Jul 30 14:49:37.948778 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:37.948745 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" event={"ID":"514622fb-2dfc-4ac3-bad9-3e596a838b82","Type":"ContainerStarted","Data":"5fb5ebbd9cb3c2a74427f19c3930e9d2f13e8c039bd3cb28160b6e8d03100477"} Jul 30 14:49:38.058853 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:38.058814 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:38.061342 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:38.061312 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3a92369c-66e2-4c03-a22d-8ef9cc74231c-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-f8s2b\" (UID: \"3a92369c-66e2-4c03-a22d-8ef9cc74231c\") " pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:38.294280 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:38.294249 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-f8s2b" Jul 30 14:49:38.468481 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:38.468450 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-f8s2b"] Jul 30 14:49:38.953482 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:38.953407 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8s2b" event={"ID":"3a92369c-66e2-4c03-a22d-8ef9cc74231c","Type":"ContainerStarted","Data":"16739fccc8f9ccb80add5f9dca31610d6d945c763c693285728000a1c97fa3a6"} Jul 30 14:49:39.958990 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:39.958949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" event={"ID":"514622fb-2dfc-4ac3-bad9-3e596a838b82","Type":"ContainerStarted","Data":"4dbd02d8824a43c2f270c0cb7bda4e120f3cc1413993eae5c906118af440d49b"} Jul 30 14:49:39.959414 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:39.959113 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" Jul 30 14:49:39.964010 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:39.963984 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" Jul 30 14:49:39.975932 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:39.975885 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-7f8d78f54b-rbnbf" podStartSLOduration=2.122708379 podStartE2EDuration="3.97587048s" totalImagesPullingTime="1.853162101s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:39.975294061 +0000 UTC m=+106.958577399" watchObservedRunningTime="2026-07-30 14:49:39.97587048 +0000 UTC m=+106.959153816" Jul 30 14:49:40.726596 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.726565 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-47ks2"] Jul 30 14:49:40.728829 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.728806 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.730859 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.730826 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 30 14:49:40.730972 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.730924 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 30 14:49:40.731471 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.731434 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-769bq\"" Jul 30 14:49:40.731570 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.731472 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 30 14:49:40.739373 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.739344 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-47ks2"] Jul 30 14:49:40.880251 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.880208 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.880251 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.880254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680eabbc-c97f-4554-979f-f8d88255f5f0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.880489 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.880289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.880489 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.880337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb5m2\" (UniqueName: \"kubernetes.io/projected/680eabbc-c97f-4554-979f-f8d88255f5f0-kube-api-access-qb5m2\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.981502 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.981422 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb5m2\" (UniqueName: \"kubernetes.io/projected/680eabbc-c97f-4554-979f-f8d88255f5f0-kube-api-access-qb5m2\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.981952 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.981518 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.981952 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.981551 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680eabbc-c97f-4554-979f-f8d88255f5f0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.981952 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.981585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.981952 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:40.981645 2564 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 30 14:49:40.981952 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:40.981726 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-tls podName:680eabbc-c97f-4554-979f-f8d88255f5f0 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:41.481706965 +0000 UTC m=+108.464990295 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-47ks2" (UID: "680eabbc-c97f-4554-979f-f8d88255f5f0") : secret "prometheus-operator-tls" not found Jul 30 14:49:40.982360 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.982340 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/680eabbc-c97f-4554-979f-f8d88255f5f0-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.984375 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.984349 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:40.992224 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:40.992196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb5m2\" (UniqueName: \"kubernetes.io/projected/680eabbc-c97f-4554-979f-f8d88255f5f0-kube-api-access-qb5m2\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:41.486821 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:41.486785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:41.489780 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:41.489754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/680eabbc-c97f-4554-979f-f8d88255f5f0-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-47ks2\" (UID: \"680eabbc-c97f-4554-979f-f8d88255f5f0\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:41.640225 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:41.640193 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" Jul 30 14:49:43.821711 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.821679 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-77bdb669d6-5cqbb"] Jul 30 14:49:43.823968 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.823943 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:43.825150 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.825127 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-47ks2"] Jul 30 14:49:43.826214 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.826194 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 30 14:49:43.826781 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.826758 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 30 14:49:43.826781 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.826775 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-m7lc8\"" Jul 30 14:49:43.826929 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.826823 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 30 14:49:43.826929 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.826757 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 30 14:49:43.827043 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.827025 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 30 14:49:43.827354 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.827335 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 30 14:49:43.827446 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.827373 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 30 14:49:43.828927 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:43.828906 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod680eabbc_c97f_4554_979f_f8d88255f5f0.slice/crio-f8d6789bd169d8d6d7e782d470ec7d0c370e4564aba32581e79e471b29a0855f WatchSource:0}: Error finding container f8d6789bd169d8d6d7e782d470ec7d0c370e4564aba32581e79e471b29a0855f: Status 404 returned error can't find the container with id f8d6789bd169d8d6d7e782d470ec7d0c370e4564aba32581e79e471b29a0855f Jul 30 14:49:43.832139 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.832122 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 30 14:49:43.835224 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.835175 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77bdb669d6-5cqbb"] Jul 30 14:49:43.904095 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.904047 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-serving-cert\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:43.904095 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.904094 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-service-ca\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:43.904330 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.904172 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-oauth-config\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:43.904330 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.904199 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxvgp\" (UniqueName: \"kubernetes.io/projected/2a5c96aa-dd65-446c-a94d-46676c55163e-kube-api-access-dxvgp\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:43.904330 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.904224 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-oauth-serving-cert\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:43.904330 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.904258 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-console-config\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:43.904488 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.904337 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-trusted-ca-bundle\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:43.979253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.979220 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8s2b" event={"ID":"3a92369c-66e2-4c03-a22d-8ef9cc74231c","Type":"ContainerStarted","Data":"b790baf4d38cc43d105182654252c45e9b200ad494946a99e4732140fdeeb822"} Jul 30 14:49:43.980302 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:43.980279 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" event={"ID":"680eabbc-c97f-4554-979f-f8d88255f5f0","Type":"ContainerStarted","Data":"f8d6789bd169d8d6d7e782d470ec7d0c370e4564aba32581e79e471b29a0855f"} Jul 30 14:49:44.004694 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.004665 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-oauth-config\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.004812 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.004706 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxvgp\" (UniqueName: \"kubernetes.io/projected/2a5c96aa-dd65-446c-a94d-46676c55163e-kube-api-access-dxvgp\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.004812 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.004734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-oauth-serving-cert\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.004812 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.004769 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-console-config\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.004812 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.004799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-trusted-ca-bundle\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.004995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.004839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-serving-cert\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.004995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.004860 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-service-ca\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.005609 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.005582 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-service-ca\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.005730 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.005661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-oauth-serving-cert\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.005730 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.005703 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-console-config\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.006008 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.005978 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-trusted-ca-bundle\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.007293 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.007274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-oauth-config\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.007453 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.007435 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-serving-cert\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.012569 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.012547 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxvgp\" (UniqueName: \"kubernetes.io/projected/2a5c96aa-dd65-446c-a94d-46676c55163e-kube-api-access-dxvgp\") pod \"console-77bdb669d6-5cqbb\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.140686 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.140648 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:44.300436 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.300404 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-77bdb669d6-5cqbb"] Jul 30 14:49:44.304095 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:44.304039 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5c96aa_dd65_446c_a94d_46676c55163e.slice/crio-9f177c0d3eaa44a93fd37a86f95a5a5bd8687fa06ee84306ffa4d7936b67faf6 WatchSource:0}: Error finding container 9f177c0d3eaa44a93fd37a86f95a5a5bd8687fa06ee84306ffa4d7936b67faf6: Status 404 returned error can't find the container with id 9f177c0d3eaa44a93fd37a86f95a5a5bd8687fa06ee84306ffa4d7936b67faf6 Jul 30 14:49:44.943551 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.942651 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-gx9sl" Jul 30 14:49:44.985771 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:44.985686 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bdb669d6-5cqbb" event={"ID":"2a5c96aa-dd65-446c-a94d-46676c55163e","Type":"ContainerStarted","Data":"9f177c0d3eaa44a93fd37a86f95a5a5bd8687fa06ee84306ffa4d7936b67faf6"} Jul 30 14:49:49.001386 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:49.001351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8s2b" event={"ID":"3a92369c-66e2-4c03-a22d-8ef9cc74231c","Type":"ContainerStarted","Data":"a862e07a85e282e06fc8fc7f34e29a745e6b4393f747c3b785cd0ae67e236219"} Jul 30 14:49:49.003085 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:49.003044 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" event={"ID":"680eabbc-c97f-4554-979f-f8d88255f5f0","Type":"ContainerStarted","Data":"dd6a3b329f8d17b2596c61b63c4506e30f9efbe84af66e8c931e8b9e9d0c7098"} Jul 30 14:49:49.003208 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:49.003114 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" event={"ID":"680eabbc-c97f-4554-979f-f8d88255f5f0","Type":"ContainerStarted","Data":"44337439957bc07a85f0c7f512220f163a1e2f272b37413b9849fcfb3b1758ca"} Jul 30 14:49:49.004571 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:49.004548 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bdb669d6-5cqbb" event={"ID":"2a5c96aa-dd65-446c-a94d-46676c55163e","Type":"ContainerStarted","Data":"962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d"} Jul 30 14:49:49.021240 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:49.021197 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-47ks2" podStartSLOduration=7.383666996 podStartE2EDuration="9.021183913s" totalImagesPullingTime="1.637516917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:49.020641184 +0000 UTC m=+116.003924519" watchObservedRunningTime="2026-07-30 14:49:49.021183913 +0000 UTC m=+116.004467247" Jul 30 14:49:49.042460 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:49.042415 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-77bdb669d6-5cqbb" podStartSLOduration=1.725108758 podStartE2EDuration="6.042401254s" totalImagesPullingTime="4.317292496s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:49.041422497 +0000 UTC m=+116.024705833" watchObservedRunningTime="2026-07-30 14:49:49.042401254 +0000 UTC m=+116.025684590" Jul 30 14:49:51.137758 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.137715 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx"] Jul 30 14:49:51.153100 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.152934 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-qtrc2"] Jul 30 14:49:51.153272 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.153149 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.155787 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.155607 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 30 14:49:51.155787 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.155610 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 30 14:49:51.156115 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.156095 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-72v7w\"" Jul 30 14:49:51.156115 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.156094 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 30 14:49:51.156790 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.156768 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx"] Jul 30 14:49:51.157111 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.156921 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.160521 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.159259 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 30 14:49:51.160521 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.159761 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 30 14:49:51.160521 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.160102 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 30 14:49:51.160521 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.160273 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-m4t2j\"" Jul 30 14:49:51.269065 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269027 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d2kd\" (UniqueName: \"kubernetes.io/projected/c11b1f6d-18f2-492e-9119-d58d7365342a-kube-api-access-7d2kd\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.269253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269092 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcc3378-f0c7-4590-80d7-ef8061034f14-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.269253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269152 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-tls\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.269253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269238 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0dcc3378-f0c7-4590-80d7-ef8061034f14-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.269409 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-root\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.269409 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269286 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c11b1f6d-18f2-492e-9119-d58d7365342a-metrics-client-ca\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.269409 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269339 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.269409 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.269409 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269379 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.269409 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269403 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.269693 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269432 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.269693 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269458 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-sys\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.269693 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269485 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dpc\" (UniqueName: \"kubernetes.io/projected/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-api-access-t6dpc\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.269693 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269543 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-textfile\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.269693 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.269586 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-wtmp\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370547 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0dcc3378-f0c7-4590-80d7-ef8061034f14-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370608 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-root\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370638 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c11b1f6d-18f2-492e-9119-d58d7365342a-metrics-client-ca\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370702 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370728 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370779 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370806 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.370816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-sys\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.371429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370855 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dpc\" (UniqueName: \"kubernetes.io/projected/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-api-access-t6dpc\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.371429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370877 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-textfile\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.371429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370905 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-wtmp\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.371429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d2kd\" (UniqueName: \"kubernetes.io/projected/c11b1f6d-18f2-492e-9119-d58d7365342a-kube-api-access-7d2kd\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.371429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370986 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcc3378-f0c7-4590-80d7-ef8061034f14-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.371429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.370984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-root\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.371429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.371015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-tls\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.371429 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.371372 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/0dcc3378-f0c7-4590-80d7-ef8061034f14-volume-directive-shadow\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.371804 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.371561 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c11b1f6d-18f2-492e-9119-d58d7365342a-metrics-client-ca\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.371804 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.371706 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-wtmp\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.371804 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.371740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.371804 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.371754 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c11b1f6d-18f2-492e-9119-d58d7365342a-sys\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.372005 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.371848 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-accelerators-collector-config\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.372060 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.372031 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-textfile\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.372601 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.372577 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0dcc3378-f0c7-4590-80d7-ef8061034f14-metrics-client-ca\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.373783 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.373740 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-tls\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.374281 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.374236 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c11b1f6d-18f2-492e-9119-d58d7365342a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.374385 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.374365 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.374720 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.374695 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-state-metrics-tls\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.380414 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.380384 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dpc\" (UniqueName: \"kubernetes.io/projected/0dcc3378-f0c7-4590-80d7-ef8061034f14-kube-api-access-t6dpc\") pod \"kube-state-metrics-56d9f944b4-9szkx\" (UID: \"0dcc3378-f0c7-4590-80d7-ef8061034f14\") " pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.380523 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.380454 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d2kd\" (UniqueName: \"kubernetes.io/projected/c11b1f6d-18f2-492e-9119-d58d7365342a-kube-api-access-7d2kd\") pod \"node-exporter-qtrc2\" (UID: \"c11b1f6d-18f2-492e-9119-d58d7365342a\") " pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:51.470263 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.470181 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" Jul 30 14:49:51.477436 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:51.477409 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-qtrc2" Jul 30 14:49:52.101552 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:52.101514 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc11b1f6d_18f2_492e_9119_d58d7365342a.slice/crio-4ff34180b25e41c2ed0b087aa72b4d205ac4e1c08ff6f1ba7cf04b56254a011a WatchSource:0}: Error finding container 4ff34180b25e41c2ed0b087aa72b4d205ac4e1c08ff6f1ba7cf04b56254a011a: Status 404 returned error can't find the container with id 4ff34180b25e41c2ed0b087aa72b4d205ac4e1c08ff6f1ba7cf04b56254a011a Jul 30 14:49:52.150858 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.150441 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 14:49:52.155471 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.154428 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162182 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162238 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pwtfm\"" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162348 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162451 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162182 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162453 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162576 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162693 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 30 14:49:52.162991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.162757 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 30 14:49:52.163529 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.163101 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 30 14:49:52.176645 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.176579 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 14:49:52.253449 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.253415 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx"] Jul 30 14:49:52.256614 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:52.256591 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dcc3378_f0c7_4590_80d7_ef8061034f14.slice/crio-85028973482ea5f0c386a65ca485bb32bb5f3b2d596e7069fbc4cc312f8d0baa WatchSource:0}: Error finding container 85028973482ea5f0c386a65ca485bb32bb5f3b2d596e7069fbc4cc312f8d0baa: Status 404 returned error can't find the container with id 85028973482ea5f0c386a65ca485bb32bb5f3b2d596e7069fbc4cc312f8d0baa Jul 30 14:49:52.278268 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278240 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278386 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278274 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278386 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278304 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278386 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278327 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278386 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278375 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278524 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278407 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dxrh\" (UniqueName: \"kubernetes.io/projected/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-kube-api-access-2dxrh\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278524 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278433 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278524 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278460 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278524 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278494 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-web-config\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278679 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278522 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278679 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278562 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278679 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.278679 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.278641 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-config-out\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.379972 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.379886 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.379972 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.379925 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dxrh\" (UniqueName: \"kubernetes.io/projected/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-kube-api-access-2dxrh\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.379972 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.379948 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.379971 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380005 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-web-config\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380029 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380062 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380111 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380141 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-config-out\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:52.380180 2564 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:52.380254 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-main-tls podName:be4a0dda-fbae-4e8e-a056-11e6d0b709b2 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:52.880233442 +0000 UTC m=+119.863516760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "be4a0dda-fbae-4e8e-a056-11e6d0b709b2") : secret "alertmanager-main-tls" not found Jul 30 14:49:52.380267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380188 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380309 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380348 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380358 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.380375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.380685 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:49:52.380447 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-trusted-ca-bundle podName:be4a0dda-fbae-4e8e-a056-11e6d0b709b2 nodeName:}" failed. No retries permitted until 2026-07-30 14:49:52.880434344 +0000 UTC m=+119.863717662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "be4a0dda-fbae-4e8e-a056-11e6d0b709b2") : configmap references non-existent config key: ca-bundle.crt Jul 30 14:49:52.381267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.381217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.383554 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.383467 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-web-config\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.383554 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.383502 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.383554 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.383545 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.383769 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.383610 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.383843 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.383778 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-config-volume\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.383843 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.383786 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-config-out\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.383843 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.383828 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-tls-assets\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.384126 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.384105 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.395944 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.395908 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dxrh\" (UniqueName: \"kubernetes.io/projected/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-kube-api-access-2dxrh\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.428378 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.428344 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-757dcf8f9-8j2qt"] Jul 30 14:49:52.432771 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.432743 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:49:52.884140 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.884106 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.884317 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.884208 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.885155 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.885106 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:52.888608 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:52.888584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/be4a0dda-fbae-4e8e-a056-11e6d0b709b2-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"be4a0dda-fbae-4e8e-a056-11e6d0b709b2\") " pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:53.028020 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.027938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-f8s2b" event={"ID":"3a92369c-66e2-4c03-a22d-8ef9cc74231c","Type":"ContainerStarted","Data":"958d39b7f580405b59e8d28f5814c251b7c9d615be186d1c50a53d621dbc3911"} Jul 30 14:49:53.029661 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.029630 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtrc2" event={"ID":"c11b1f6d-18f2-492e-9119-d58d7365342a","Type":"ContainerStarted","Data":"4ff34180b25e41c2ed0b087aa72b4d205ac4e1c08ff6f1ba7cf04b56254a011a"} Jul 30 14:49:53.030980 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.030955 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" event={"ID":"0dcc3378-f0c7-4590-80d7-ef8061034f14","Type":"ContainerStarted","Data":"85028973482ea5f0c386a65ca485bb32bb5f3b2d596e7069fbc4cc312f8d0baa"} Jul 30 14:49:53.050005 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.049943 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-f8s2b" podStartSLOduration=33.740168039 podStartE2EDuration="47.049928579s" totalImagesPullingTime="13.30976054s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 14:49:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:53.048284742 +0000 UTC m=+120.031568078" watchObservedRunningTime="2026-07-30 14:49:53.049928579 +0000 UTC m=+120.033211914" Jul 30 14:49:53.072228 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.072200 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 30 14:49:53.109550 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.109515 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h"] Jul 30 14:49:53.113560 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.113538 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.115803 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.115766 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" Jul 30 14:49:53.116505 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.116485 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" Jul 30 14:49:53.116627 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.116606 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" Jul 30 14:49:53.116821 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.116801 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-4tjd9aigq2g0t\"" Jul 30 14:49:53.116899 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.116840 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" Jul 30 14:49:53.117168 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.117146 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" Jul 30 14:49:53.117383 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.117362 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v5x9h\"" Jul 30 14:49:53.130630 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.130582 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h"] Jul 30 14:49:53.227884 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.227859 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 30 14:49:53.229906 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:53.229878 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe4a0dda_fbae_4e8e_a056_11e6d0b709b2.slice/crio-f9037bf4177aee1c8b2ceece92ff99f4a6586f316ba520efb0369203d1b07226 WatchSource:0}: Error finding container f9037bf4177aee1c8b2ceece92ff99f4a6586f316ba520efb0369203d1b07226: Status 404 returned error can't find the container with id f9037bf4177aee1c8b2ceece92ff99f4a6586f316ba520efb0369203d1b07226 Jul 30 14:49:53.287861 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.287831 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.288009 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.287871 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-metrics-client-ca\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.288009 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.287945 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.288123 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.288061 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghgsd\" (UniqueName: \"kubernetes.io/projected/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-kube-api-access-ghgsd\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.288171 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.288150 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.288238 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.288215 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-tls\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.288301 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.288287 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-grpc-tls\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.288379 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.288360 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.389605 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.389521 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.389605 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.389568 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghgsd\" (UniqueName: \"kubernetes.io/projected/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-kube-api-access-ghgsd\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.389828 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.389609 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.389828 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.389639 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-tls\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.389828 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.389675 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-grpc-tls\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.389828 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.389739 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.389828 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.389772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.390060 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.389841 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-metrics-client-ca\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.390689 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.390658 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-metrics-client-ca\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.392259 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.392234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.392419 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.392374 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.392690 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.392669 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-tls\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.392770 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.392750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.392818 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.392800 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-grpc-tls\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.392852 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.392819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.398726 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.398702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghgsd\" (UniqueName: \"kubernetes.io/projected/89b12dec-2cf1-46a3-b49f-4618aaa3df6b-kube-api-access-ghgsd\") pod \"thanos-querier-7b9d755bf9-k4c7h\" (UID: \"89b12dec-2cf1-46a3-b49f-4618aaa3df6b\") " pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.427866 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.427825 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-v5x9h\"" Jul 30 14:49:53.436253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.436204 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:53.600104 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:53.599977 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h"] Jul 30 14:49:53.604186 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:53.604155 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b12dec_2cf1_46a3_b49f_4618aaa3df6b.slice/crio-408c27d06d302119e4cf8d9b0754cbd28f21c1e5d455cbe4dd8f9839d389d351 WatchSource:0}: Error finding container 408c27d06d302119e4cf8d9b0754cbd28f21c1e5d455cbe4dd8f9839d389d351: Status 404 returned error can't find the container with id 408c27d06d302119e4cf8d9b0754cbd28f21c1e5d455cbe4dd8f9839d389d351 Jul 30 14:49:54.036205 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:54.036162 2564 generic.go:356] "Generic (PLEG): container finished" podID="c11b1f6d-18f2-492e-9119-d58d7365342a" containerID="5f2220e644d8361cba7276f43ab6a02a3e68603e0c10444c83cf975d03bbcdea" exitCode=0 Jul 30 14:49:54.036401 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:54.036256 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtrc2" event={"ID":"c11b1f6d-18f2-492e-9119-d58d7365342a","Type":"ContainerDied","Data":"5f2220e644d8361cba7276f43ab6a02a3e68603e0c10444c83cf975d03bbcdea"} Jul 30 14:49:54.038220 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:54.038195 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be4a0dda-fbae-4e8e-a056-11e6d0b709b2","Type":"ContainerStarted","Data":"f9037bf4177aee1c8b2ceece92ff99f4a6586f316ba520efb0369203d1b07226"} Jul 30 14:49:54.039327 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:54.039191 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" event={"ID":"89b12dec-2cf1-46a3-b49f-4618aaa3df6b","Type":"ContainerStarted","Data":"408c27d06d302119e4cf8d9b0754cbd28f21c1e5d455cbe4dd8f9839d389d351"} Jul 30 14:49:54.141251 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:54.141226 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:54.141360 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:54.141318 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:54.146988 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:54.146964 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:55.044892 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.044861 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtrc2" event={"ID":"c11b1f6d-18f2-492e-9119-d58d7365342a","Type":"ContainerStarted","Data":"2b396fdc3c31fe38ad50200e2d3d80edab03023c8769db0c71625a1d92145ba3"} Jul 30 14:49:55.045317 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.044911 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-qtrc2" event={"ID":"c11b1f6d-18f2-492e-9119-d58d7365342a","Type":"ContainerStarted","Data":"ffe764dd86b66e70dc5ba62a4bae161839be34e55cecb9c000b4e8d681d6b087"} Jul 30 14:49:55.046367 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.046341 2564 generic.go:356] "Generic (PLEG): container finished" podID="be4a0dda-fbae-4e8e-a056-11e6d0b709b2" containerID="980d2dbf5bd521e1abc39aa1039a46be68c258a103b0e8bd754cf53943a334f0" exitCode=0 Jul 30 14:49:55.046483 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.046382 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be4a0dda-fbae-4e8e-a056-11e6d0b709b2","Type":"ContainerDied","Data":"980d2dbf5bd521e1abc39aa1039a46be68c258a103b0e8bd754cf53943a334f0"} Jul 30 14:49:55.048689 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.048666 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" event={"ID":"0dcc3378-f0c7-4590-80d7-ef8061034f14","Type":"ContainerStarted","Data":"c38bc45d6cf130faaa4a204dec4a4a3a6df0c7473f1c3573c512c98d6035bbe9"} Jul 30 14:49:55.048793 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.048720 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" event={"ID":"0dcc3378-f0c7-4590-80d7-ef8061034f14","Type":"ContainerStarted","Data":"c02439429dc472589bfb01e613c43a1a15555fc451f8053a470ec102e8c034f9"} Jul 30 14:49:55.048793 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.048740 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" event={"ID":"0dcc3378-f0c7-4590-80d7-ef8061034f14","Type":"ContainerStarted","Data":"7db2d2e11fbe3b1e060531cc57b6a42e23fa242502e76753f4295a7e30ccbc4d"} Jul 30 14:49:55.052445 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.052424 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:49:55.064819 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.064776 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-qtrc2" podStartSLOduration=2.9827977900000002 podStartE2EDuration="4.064761565s" totalImagesPullingTime="1.010748496s" totalInitContainerRuntime="71.215279ms" isStatefulPod=true podCreationTimestamp="2026-07-30 14:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:55.06367594 +0000 UTC m=+122.046959277" watchObservedRunningTime="2026-07-30 14:49:55.064761565 +0000 UTC m=+122.048044900" Jul 30 14:49:55.130653 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:55.130551 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56d9f944b4-9szkx" podStartSLOduration=2.30989509 podStartE2EDuration="4.130529274s" totalImagesPullingTime="1.820634184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:55.129606042 +0000 UTC m=+122.112889393" watchObservedRunningTime="2026-07-30 14:49:55.130529274 +0000 UTC m=+122.113812610" Jul 30 14:49:56.055353 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.055309 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" event={"ID":"89b12dec-2cf1-46a3-b49f-4618aaa3df6b","Type":"ContainerStarted","Data":"ae6aae40c9fe59b8ba00748fbb757142a841272fb7fa22594d23985bcd9ca8bc"} Jul 30 14:49:56.055763 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.055426 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" event={"ID":"89b12dec-2cf1-46a3-b49f-4618aaa3df6b","Type":"ContainerStarted","Data":"467db5204a2f846909991ca4c9a4eea10fb86562802965cfdbfb1a2bc5fce4fa"} Jul 30 14:49:56.055763 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.055467 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" event={"ID":"89b12dec-2cf1-46a3-b49f-4618aaa3df6b","Type":"ContainerStarted","Data":"d46fe7fae12baff38d502e22a33f34bc8b7c1799cb549a2315def57a5ba66c39"} Jul 30 14:49:56.298391 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.298358 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-69d6989cf7-6hrpg"] Jul 30 14:49:56.301934 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.301902 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.304583 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.304563 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Jul 30 14:49:56.304829 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.304811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Jul 30 14:49:56.304829 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.304811 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Jul 30 14:49:56.304984 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.304830 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Jul 30 14:49:56.305479 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.305456 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Jul 30 14:49:56.305720 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.305703 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-986jh\"" Jul 30 14:49:56.311909 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.311886 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\"" Jul 30 14:49:56.315772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.315606 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-serving-certs-ca-bundle\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.315772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.315672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.315772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.315708 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-secret-telemeter-client\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.315772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.315736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcgqd\" (UniqueName: \"kubernetes.io/projected/1a206462-f46b-4af4-9775-d45ffcd98de1-kube-api-access-fcgqd\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.316057 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.315823 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.316057 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.315906 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-telemeter-client-tls\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.316057 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.315989 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-metrics-client-ca\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.316057 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.316029 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-federate-client-tls\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.317576 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.317557 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69d6989cf7-6hrpg"] Jul 30 14:49:56.416496 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.416464 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-federate-client-tls\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.416669 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.416555 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-serving-certs-ca-bundle\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.416669 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.416602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.416669 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.416632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-secret-telemeter-client\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.416669 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.416663 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcgqd\" (UniqueName: \"kubernetes.io/projected/1a206462-f46b-4af4-9775-d45ffcd98de1-kube-api-access-fcgqd\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.416878 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.416694 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.416878 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.416726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-telemeter-client-tls\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.416975 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.416920 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-metrics-client-ca\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.417421 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.417391 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-serving-certs-ca-bundle\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.417879 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.417855 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-metrics-client-ca\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.417961 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.417883 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1a206462-f46b-4af4-9775-d45ffcd98de1-telemeter-trusted-ca-bundle\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.419495 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.419470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-secret-telemeter-client\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.419626 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.419601 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.420049 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.420029 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-telemeter-client-tls\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.420165 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.420151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/1a206462-f46b-4af4-9775-d45ffcd98de1-federate-client-tls\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.425792 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.425772 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcgqd\" (UniqueName: \"kubernetes.io/projected/1a206462-f46b-4af4-9775-d45ffcd98de1-kube-api-access-fcgqd\") pod \"telemeter-client-69d6989cf7-6hrpg\" (UID: \"1a206462-f46b-4af4-9775-d45ffcd98de1\") " pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.614258 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.614170 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" Jul 30 14:49:56.776740 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:56.776652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-69d6989cf7-6hrpg"] Jul 30 14:49:56.865216 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:56.865149 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a206462_f46b_4af4_9775_d45ffcd98de1.slice/crio-b8cc630cd6296d53b8346cd158501c71489a28713f7d02b52d8f410ebed6a3d7 WatchSource:0}: Error finding container b8cc630cd6296d53b8346cd158501c71489a28713f7d02b52d8f410ebed6a3d7: Status 404 returned error can't find the container with id b8cc630cd6296d53b8346cd158501c71489a28713f7d02b52d8f410ebed6a3d7 Jul 30 14:49:57.060808 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.060759 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" event={"ID":"1a206462-f46b-4af4-9775-d45ffcd98de1","Type":"ContainerStarted","Data":"b8cc630cd6296d53b8346cd158501c71489a28713f7d02b52d8f410ebed6a3d7"} Jul 30 14:49:57.450576 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.450535 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55bb6c659f-75rg7"] Jul 30 14:49:57.455549 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.455420 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.470176 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.470143 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bb6c659f-75rg7"] Jul 30 14:49:57.528850 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.528577 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-service-ca\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.528850 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.528626 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-console-config\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.528850 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.528649 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-trusted-ca-bundle\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.528850 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.528689 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-oauth-serving-cert\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.528850 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.528723 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-serving-cert\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.528850 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.528799 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7frxp\" (UniqueName: \"kubernetes.io/projected/78076b73-933a-4fc4-a9bf-7e09806f9897-kube-api-access-7frxp\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.529223 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.528934 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-oauth-config\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.629852 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.629815 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-oauth-config\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.629987 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.629876 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-service-ca\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.629987 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.629915 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-console-config\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.629987 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.629938 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-trusted-ca-bundle\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.629987 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.629982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-oauth-serving-cert\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.630217 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.630021 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-serving-cert\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.630217 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.630124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7frxp\" (UniqueName: \"kubernetes.io/projected/78076b73-933a-4fc4-a9bf-7e09806f9897-kube-api-access-7frxp\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.631219 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.631170 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-console-config\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.631387 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.631334 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-oauth-serving-cert\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.631486 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.631403 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-trusted-ca-bundle\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.631486 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.631421 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-service-ca\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.634064 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.634042 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-oauth-config\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.634408 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.634387 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-serving-cert\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.648926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.648866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7frxp\" (UniqueName: \"kubernetes.io/projected/78076b73-933a-4fc4-a9bf-7e09806f9897-kube-api-access-7frxp\") pod \"console-55bb6c659f-75rg7\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.784734 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.784649 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:49:57.947584 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:57.947550 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55bb6c659f-75rg7"] Jul 30 14:49:57.951211 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:49:57.951185 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78076b73_933a_4fc4_a9bf_7e09806f9897.slice/crio-0e02e6d5f368ef831b5428bb4bfc034c3d7b2b0634dc2f662c2b1a5863149663 WatchSource:0}: Error finding container 0e02e6d5f368ef831b5428bb4bfc034c3d7b2b0634dc2f662c2b1a5863149663: Status 404 returned error can't find the container with id 0e02e6d5f368ef831b5428bb4bfc034c3d7b2b0634dc2f662c2b1a5863149663 Jul 30 14:49:58.072715 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.072625 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be4a0dda-fbae-4e8e-a056-11e6d0b709b2","Type":"ContainerStarted","Data":"83f42c8e7078a062030c5683db6c5fd8a337530354f2ce4b46bb437aefdbbe66"} Jul 30 14:49:58.073175 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.072717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be4a0dda-fbae-4e8e-a056-11e6d0b709b2","Type":"ContainerStarted","Data":"6e78d021607b68496554fa1800d5e050a454203266840e2140ebb59c878e07e3"} Jul 30 14:49:58.073175 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.072758 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be4a0dda-fbae-4e8e-a056-11e6d0b709b2","Type":"ContainerStarted","Data":"9dcb112c98010b1417c1ecad86fc6311c83bb98f4414c99a01cb1bd3b22def52"} Jul 30 14:49:58.073175 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.072797 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be4a0dda-fbae-4e8e-a056-11e6d0b709b2","Type":"ContainerStarted","Data":"5b01158b714e614bb4cb56a9e43d87efdf96faf8746a63581f70000035675a3d"} Jul 30 14:49:58.073175 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.072833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be4a0dda-fbae-4e8e-a056-11e6d0b709b2","Type":"ContainerStarted","Data":"e3d5555d9011b7dab8d818f983d1974378ffc50bfbb657d2fb78cd893398a9f5"} Jul 30 14:49:58.073175 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.072870 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"be4a0dda-fbae-4e8e-a056-11e6d0b709b2","Type":"ContainerStarted","Data":"91595610cdedc7975863614968d2900ad1ad03fef0a6febdceeeedb2cdb7511a"} Jul 30 14:49:58.077595 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.077564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" event={"ID":"89b12dec-2cf1-46a3-b49f-4618aaa3df6b","Type":"ContainerStarted","Data":"226710eed835a7bbf80b777b5b0f838455310dc62120eb57c981068ecd4ad664"} Jul 30 14:49:58.077738 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.077647 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" event={"ID":"89b12dec-2cf1-46a3-b49f-4618aaa3df6b","Type":"ContainerStarted","Data":"34f04cebbb13407538a3655389c9d2ed027e3497693fa494261b2613431a40ba"} Jul 30 14:49:58.077738 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.077687 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" event={"ID":"89b12dec-2cf1-46a3-b49f-4618aaa3df6b","Type":"ContainerStarted","Data":"d5b97b55282499d5bcdc4fe72cfcca53526166c27badf0372e9ce7a4f3adec7d"} Jul 30 14:49:58.078116 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.078022 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:49:58.079858 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.079813 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bb6c659f-75rg7" event={"ID":"78076b73-933a-4fc4-a9bf-7e09806f9897","Type":"ContainerStarted","Data":"c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5"} Jul 30 14:49:58.079983 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.079871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bb6c659f-75rg7" event={"ID":"78076b73-933a-4fc4-a9bf-7e09806f9897","Type":"ContainerStarted","Data":"0e02e6d5f368ef831b5428bb4bfc034c3d7b2b0634dc2f662c2b1a5863149663"} Jul 30 14:49:58.107842 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.107770 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.197506899 podStartE2EDuration="6.107751772s" totalImagesPullingTime="3.890736246s" totalInitContainerRuntime="19.508627ms" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:52 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:58.105035748 +0000 UTC m=+125.088319083" watchObservedRunningTime="2026-07-30 14:49:58.107751772 +0000 UTC m=+125.091035109" Jul 30 14:49:58.139932 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.139866 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" podStartSLOduration=1.6593921919999999 podStartE2EDuration="5.139833055s" totalImagesPullingTime="3.480440863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:58.139448187 +0000 UTC m=+125.122731546" watchObservedRunningTime="2026-07-30 14:49:58.139833055 +0000 UTC m=+125.123116393" Jul 30 14:49:58.165448 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:58.165387 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55bb6c659f-75rg7" podStartSLOduration=1.165366215 podStartE2EDuration="1.165366215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:49:58.164489366 +0000 UTC m=+125.147772705" watchObservedRunningTime="2026-07-30 14:49:58.165366215 +0000 UTC m=+125.148649547" Jul 30 14:49:59.088203 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:59.088150 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" event={"ID":"1a206462-f46b-4af4-9775-d45ffcd98de1","Type":"ContainerStarted","Data":"1119c3945b8061f2b1a8dcf8c459b155d2f4c788a4fedfd6376bfbb28aa77297"} Jul 30 14:49:59.088595 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:49:59.088260 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" event={"ID":"1a206462-f46b-4af4-9775-d45ffcd98de1","Type":"ContainerStarted","Data":"752b1f74d680eb7844ef9eae2dd0297bf2422e67342f4256928ed568fbbdc9b9"} Jul 30 14:50:00.096740 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:00.096707 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" event={"ID":"1a206462-f46b-4af4-9775-d45ffcd98de1","Type":"ContainerStarted","Data":"f6ec5a1327990d7e07dca23de7a4c4b66faddc47ae420b6e36dd2aaea526a32e"} Jul 30 14:50:00.153023 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:00.152969 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-69d6989cf7-6hrpg" podStartSLOduration=2.086353612 podStartE2EDuration="4.152951279s" totalImagesPullingTime="2.066597667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:49:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:50:00.151246996 +0000 UTC m=+127.134530335" watchObservedRunningTime="2026-07-30 14:50:00.152951279 +0000 UTC m=+127.136234614" Jul 30 14:50:01.054462 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.054428 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bb6c659f-75rg7"] Jul 30 14:50:01.093947 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.093910 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-645bdb6d5b-lv9sx"] Jul 30 14:50:01.100045 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.100012 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.115289 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.115259 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645bdb6d5b-lv9sx"] Jul 30 14:50:01.163860 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.163828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-serving-cert\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.163860 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.163864 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-service-ca\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.164135 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.163944 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-oauth-serving-cert\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.164135 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.164003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-console-config\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.164236 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.164177 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-oauth-config\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.164326 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.164311 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-trusted-ca-bundle\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.164375 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.164338 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqfgd\" (UniqueName: \"kubernetes.io/projected/802e3adb-22fb-4935-bc9e-6caccba559a1-kube-api-access-gqfgd\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.265276 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.265234 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-oauth-serving-cert\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.265276 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.265276 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-console-config\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.265509 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.265317 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-oauth-config\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.265509 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.265366 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-trusted-ca-bundle\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.265509 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.265423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqfgd\" (UniqueName: \"kubernetes.io/projected/802e3adb-22fb-4935-bc9e-6caccba559a1-kube-api-access-gqfgd\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.265509 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.265495 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-serving-cert\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.265700 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.265530 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-service-ca\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.266106 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.266054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-oauth-serving-cert\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.266236 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.266125 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-console-config\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.266310 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.266176 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-service-ca\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.266419 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.266398 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-trusted-ca-bundle\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.267755 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.267733 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-oauth-config\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.267998 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.267980 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-serving-cert\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.274579 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.274548 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqfgd\" (UniqueName: \"kubernetes.io/projected/802e3adb-22fb-4935-bc9e-6caccba559a1-kube-api-access-gqfgd\") pod \"console-645bdb6d5b-lv9sx\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.418331 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.418238 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:01.562142 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:01.562108 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-645bdb6d5b-lv9sx"] Jul 30 14:50:01.568626 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:50:01.568598 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod802e3adb_22fb_4935_bc9e_6caccba559a1.slice/crio-eaa5369aa52fb357cc7a45049dbf14586feabb548855a2379028323a22816f5e WatchSource:0}: Error finding container eaa5369aa52fb357cc7a45049dbf14586feabb548855a2379028323a22816f5e: Status 404 returned error can't find the container with id eaa5369aa52fb357cc7a45049dbf14586feabb548855a2379028323a22816f5e Jul 30 14:50:02.112396 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:02.112351 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645bdb6d5b-lv9sx" event={"ID":"802e3adb-22fb-4935-bc9e-6caccba559a1","Type":"ContainerStarted","Data":"e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91"} Jul 30 14:50:02.112799 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:02.112419 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645bdb6d5b-lv9sx" event={"ID":"802e3adb-22fb-4935-bc9e-6caccba559a1","Type":"ContainerStarted","Data":"eaa5369aa52fb357cc7a45049dbf14586feabb548855a2379028323a22816f5e"} Jul 30 14:50:02.134424 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:02.134371 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-645bdb6d5b-lv9sx" podStartSLOduration=1.1343512 podStartE2EDuration="1.1343512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:50:02.132711409 +0000 UTC m=+129.115994745" watchObservedRunningTime="2026-07-30 14:50:02.1343512 +0000 UTC m=+129.117634535" Jul 30 14:50:02.376693 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:02.376599 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:50:02.378988 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:02.378964 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71-metrics-certs\") pod \"network-metrics-daemon-4k86d\" (UID: \"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71\") " pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:50:02.495666 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:02.495631 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-82hs8\"" Jul 30 14:50:02.504328 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:02.504295 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4k86d" Jul 30 14:50:02.647567 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:02.647541 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4k86d"] Jul 30 14:50:02.650052 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:50:02.650025 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbf6dd4f_65c5_4b5f_b847_b11eabbf1d71.slice/crio-a060c5b3cfb7159b160addf862a8b3c5d5b37dc5e1ddf3f1a4a2cf29153933d2 WatchSource:0}: Error finding container a060c5b3cfb7159b160addf862a8b3c5d5b37dc5e1ddf3f1a4a2cf29153933d2: Status 404 returned error can't find the container with id a060c5b3cfb7159b160addf862a8b3c5d5b37dc5e1ddf3f1a4a2cf29153933d2 Jul 30 14:50:03.119125 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:03.119063 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4k86d" event={"ID":"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71","Type":"ContainerStarted","Data":"a060c5b3cfb7159b160addf862a8b3c5d5b37dc5e1ddf3f1a4a2cf29153933d2"} Jul 30 14:50:04.097460 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:04.097429 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-7b9d755bf9-k4c7h" Jul 30 14:50:04.127545 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:04.127501 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4k86d" event={"ID":"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71","Type":"ContainerStarted","Data":"28bf2a655150bfb6a8598021b80e26fc50190f6492f9150f4222dd243b438128"} Jul 30 14:50:04.128033 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:04.127591 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4k86d" event={"ID":"bbf6dd4f-65c5-4b5f-b847-b11eabbf1d71","Type":"ContainerStarted","Data":"40bf2f1b43f55b589a8a8c2c89d5229025027897b1675ad07298ccf2b7bb74b1"} Jul 30 14:50:04.149403 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:04.149347 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4k86d" podStartSLOduration=129.118698462 podStartE2EDuration="2m10.149329515s" totalImagesPullingTime="1.030631053s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:50:04.148182255 +0000 UTC m=+131.131465591" watchObservedRunningTime="2026-07-30 14:50:04.149329515 +0000 UTC m=+131.132612851" Jul 30 14:50:07.785584 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:07.785545 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:50:11.418397 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:11.418361 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:11.418903 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:11.418445 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:11.423251 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:11.423229 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:12.164316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:12.164289 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:50:12.213467 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:12.213429 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77bdb669d6-5cqbb"] Jul 30 14:50:17.447246 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.447184 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" podUID="4253f459-2c3c-427f-b9da-3449dbd7212f" containerName="registry" containerID="cri-o://c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa" gracePeriod=30 Jul 30 14:50:17.679782 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.679757 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:50:17.813897 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.813852 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-bound-sa-token\") pod \"4253f459-2c3c-427f-b9da-3449dbd7212f\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " Jul 30 14:50:17.814121 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.813921 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4253f459-2c3c-427f-b9da-3449dbd7212f-ca-trust-extracted\") pod \"4253f459-2c3c-427f-b9da-3449dbd7212f\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " Jul 30 14:50:17.814121 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.813944 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-certificates\") pod \"4253f459-2c3c-427f-b9da-3449dbd7212f\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " Jul 30 14:50:17.814121 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.813962 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") pod \"4253f459-2c3c-427f-b9da-3449dbd7212f\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " Jul 30 14:50:17.814121 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.813992 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpn99\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-kube-api-access-lpn99\") pod \"4253f459-2c3c-427f-b9da-3449dbd7212f\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " Jul 30 14:50:17.814347 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.814123 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-installation-pull-secrets\") pod \"4253f459-2c3c-427f-b9da-3449dbd7212f\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " Jul 30 14:50:17.814347 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.814163 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-trusted-ca\") pod \"4253f459-2c3c-427f-b9da-3449dbd7212f\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " Jul 30 14:50:17.814347 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.814201 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-image-registry-private-configuration\") pod \"4253f459-2c3c-427f-b9da-3449dbd7212f\" (UID: \"4253f459-2c3c-427f-b9da-3449dbd7212f\") " Jul 30 14:50:17.814551 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.814514 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4253f459-2c3c-427f-b9da-3449dbd7212f" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:17.814739 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.814711 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4253f459-2c3c-427f-b9da-3449dbd7212f" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:17.816539 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.816510 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4253f459-2c3c-427f-b9da-3449dbd7212f" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:50:17.816769 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.816721 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4253f459-2c3c-427f-b9da-3449dbd7212f" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:50:17.816900 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.816881 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4253f459-2c3c-427f-b9da-3449dbd7212f" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:50:17.816964 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.816935 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4253f459-2c3c-427f-b9da-3449dbd7212f" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:50:17.816964 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.816943 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-kube-api-access-lpn99" (OuterVolumeSpecName: "kube-api-access-lpn99") pod "4253f459-2c3c-427f-b9da-3449dbd7212f" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f"). InnerVolumeSpecName "kube-api-access-lpn99". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:50:17.822590 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.822557 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4253f459-2c3c-427f-b9da-3449dbd7212f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4253f459-2c3c-427f-b9da-3449dbd7212f" (UID: "4253f459-2c3c-427f-b9da-3449dbd7212f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:50:17.915263 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.915225 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpn99\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-kube-api-access-lpn99\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:17.915263 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.915255 2564 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-installation-pull-secrets\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:17.915263 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.915266 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-trusted-ca\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:17.915263 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.915276 2564 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4253f459-2c3c-427f-b9da-3449dbd7212f-image-registry-private-configuration\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:17.915531 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.915286 2564 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-bound-sa-token\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:17.915531 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.915295 2564 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4253f459-2c3c-427f-b9da-3449dbd7212f-ca-trust-extracted\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:17.915531 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.915304 2564 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-certificates\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:17.915531 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:17.915320 2564 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4253f459-2c3c-427f-b9da-3449dbd7212f-registry-tls\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:18.187353 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.187261 2564 generic.go:356] "Generic (PLEG): container finished" podID="4253f459-2c3c-427f-b9da-3449dbd7212f" containerID="c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa" exitCode=0 Jul 30 14:50:18.187353 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.187309 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" Jul 30 14:50:18.187353 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.187330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" event={"ID":"4253f459-2c3c-427f-b9da-3449dbd7212f","Type":"ContainerDied","Data":"c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa"} Jul 30 14:50:18.187598 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.187380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-757dcf8f9-8j2qt" event={"ID":"4253f459-2c3c-427f-b9da-3449dbd7212f","Type":"ContainerDied","Data":"d0954975813ab292586e30edfca4fd9d4c69f2aa8e81aeef800df7b16bf1eb6a"} Jul 30 14:50:18.187598 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.187420 2564 scope.go:117] "RemoveContainer" containerID="c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa" Jul 30 14:50:18.196436 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.196418 2564 scope.go:117] "RemoveContainer" containerID="c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa" Jul 30 14:50:18.196701 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:50:18.196680 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa\": container with ID starting with c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa not found: ID does not exist" containerID="c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa" Jul 30 14:50:18.196772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.196709 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa"} err="failed to get container status \"c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa\": rpc error: code = NotFound desc = could not find container \"c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa\": container with ID starting with c3f3ec2aea2c33bfcfe9f0fad22fe938cdb6528a88d39f6f67c64e3ce78a34fa not found: ID does not exist" Jul 30 14:50:18.207234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.207205 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-757dcf8f9-8j2qt"] Jul 30 14:50:18.212537 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:18.212516 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-757dcf8f9-8j2qt"] Jul 30 14:50:19.489083 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:19.489039 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4253f459-2c3c-427f-b9da-3449dbd7212f" path="/var/lib/kubelet/pods/4253f459-2c3c-427f-b9da-3449dbd7212f/volumes" Jul 30 14:50:26.073636 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.073570 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55bb6c659f-75rg7" podUID="78076b73-933a-4fc4-a9bf-7e09806f9897" containerName="console" containerID="cri-o://c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5" gracePeriod=15 Jul 30 14:50:26.853331 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.853305 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:50:26.996463 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996440 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-service-ca\") pod \"78076b73-933a-4fc4-a9bf-7e09806f9897\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " Jul 30 14:50:26.996572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996470 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-oauth-serving-cert\") pod \"78076b73-933a-4fc4-a9bf-7e09806f9897\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " Jul 30 14:50:26.996572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996495 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-trusted-ca-bundle\") pod \"78076b73-933a-4fc4-a9bf-7e09806f9897\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " Jul 30 14:50:26.996572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996541 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-serving-cert\") pod \"78076b73-933a-4fc4-a9bf-7e09806f9897\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " Jul 30 14:50:26.996572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996568 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-oauth-config\") pod \"78076b73-933a-4fc4-a9bf-7e09806f9897\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " Jul 30 14:50:26.996772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996603 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7frxp\" (UniqueName: \"kubernetes.io/projected/78076b73-933a-4fc4-a9bf-7e09806f9897-kube-api-access-7frxp\") pod \"78076b73-933a-4fc4-a9bf-7e09806f9897\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " Jul 30 14:50:26.996772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996640 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-console-config\") pod \"78076b73-933a-4fc4-a9bf-7e09806f9897\" (UID: \"78076b73-933a-4fc4-a9bf-7e09806f9897\") " Jul 30 14:50:26.996933 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996888 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "78076b73-933a-4fc4-a9bf-7e09806f9897" (UID: "78076b73-933a-4fc4-a9bf-7e09806f9897"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:26.997150 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.996956 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-service-ca" (OuterVolumeSpecName: "service-ca") pod "78076b73-933a-4fc4-a9bf-7e09806f9897" (UID: "78076b73-933a-4fc4-a9bf-7e09806f9897"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:26.997213 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.997139 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "78076b73-933a-4fc4-a9bf-7e09806f9897" (UID: "78076b73-933a-4fc4-a9bf-7e09806f9897"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:26.997213 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.997179 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-console-config" (OuterVolumeSpecName: "console-config") pod "78076b73-933a-4fc4-a9bf-7e09806f9897" (UID: "78076b73-933a-4fc4-a9bf-7e09806f9897"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:26.998959 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.998926 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "78076b73-933a-4fc4-a9bf-7e09806f9897" (UID: "78076b73-933a-4fc4-a9bf-7e09806f9897"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:50:26.998959 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.998950 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "78076b73-933a-4fc4-a9bf-7e09806f9897" (UID: "78076b73-933a-4fc4-a9bf-7e09806f9897"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:50:26.999099 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:26.998936 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78076b73-933a-4fc4-a9bf-7e09806f9897-kube-api-access-7frxp" (OuterVolumeSpecName: "kube-api-access-7frxp") pod "78076b73-933a-4fc4-a9bf-7e09806f9897" (UID: "78076b73-933a-4fc4-a9bf-7e09806f9897"). InnerVolumeSpecName "kube-api-access-7frxp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:50:27.098233 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.098197 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-service-ca\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:27.098233 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.098226 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-oauth-serving-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:27.098233 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.098236 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-trusted-ca-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:27.098659 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.098246 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-serving-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:27.098659 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.098256 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/78076b73-933a-4fc4-a9bf-7e09806f9897-console-oauth-config\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:27.098659 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.098264 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7frxp\" (UniqueName: \"kubernetes.io/projected/78076b73-933a-4fc4-a9bf-7e09806f9897-kube-api-access-7frxp\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:27.098659 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.098272 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/78076b73-933a-4fc4-a9bf-7e09806f9897-console-config\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:27.225262 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.225228 2564 generic.go:356] "Generic (PLEG): container finished" podID="78076b73-933a-4fc4-a9bf-7e09806f9897" containerID="c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5" exitCode=0 Jul 30 14:50:27.225430 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.225273 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55bb6c659f-75rg7" Jul 30 14:50:27.225430 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.225275 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bb6c659f-75rg7" event={"ID":"78076b73-933a-4fc4-a9bf-7e09806f9897","Type":"ContainerDied","Data":"c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5"} Jul 30 14:50:27.225430 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.225333 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55bb6c659f-75rg7" event={"ID":"78076b73-933a-4fc4-a9bf-7e09806f9897","Type":"ContainerDied","Data":"0e02e6d5f368ef831b5428bb4bfc034c3d7b2b0634dc2f662c2b1a5863149663"} Jul 30 14:50:27.225430 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.225356 2564 scope.go:117] "RemoveContainer" containerID="c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5" Jul 30 14:50:27.227158 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.227131 2564 generic.go:356] "Generic (PLEG): container finished" podID="ade968f1-c7b5-45c7-b939-a7b2213bd06b" containerID="b5080ef956ad89f159c9505f9abfe2e3e8f581483ac8df0515c2913598aa80fb" exitCode=0 Jul 30 14:50:27.227278 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.227178 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" event={"ID":"ade968f1-c7b5-45c7-b939-a7b2213bd06b","Type":"ContainerDied","Data":"b5080ef956ad89f159c9505f9abfe2e3e8f581483ac8df0515c2913598aa80fb"} Jul 30 14:50:27.227511 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.227495 2564 scope.go:117] "RemoveContainer" containerID="b5080ef956ad89f159c9505f9abfe2e3e8f581483ac8df0515c2913598aa80fb" Jul 30 14:50:27.237940 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.237918 2564 scope.go:117] "RemoveContainer" containerID="c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5" Jul 30 14:50:27.238266 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:50:27.238241 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5\": container with ID starting with c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5 not found: ID does not exist" containerID="c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5" Jul 30 14:50:27.238348 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.238278 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5"} err="failed to get container status \"c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5\": rpc error: code = NotFound desc = could not find container \"c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5\": container with ID starting with c5a80241d7ce7ee57df9f1eb0377b82fc83eea54115434c9cf843f8dc5d1e8b5 not found: ID does not exist" Jul 30 14:50:27.260657 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.260589 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55bb6c659f-75rg7"] Jul 30 14:50:27.268017 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.267980 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55bb6c659f-75rg7"] Jul 30 14:50:27.490924 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:27.490864 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78076b73-933a-4fc4-a9bf-7e09806f9897" path="/var/lib/kubelet/pods/78076b73-933a-4fc4-a9bf-7e09806f9897/volumes" Jul 30 14:50:28.234738 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:28.234703 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-vkj52" event={"ID":"ade968f1-c7b5-45c7-b939-a7b2213bd06b","Type":"ContainerStarted","Data":"93ce263182b4477943c3c02b1765ee0192799f5e1c5b2c65479a47be22b63cc4"} Jul 30 14:50:37.232827 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.232765 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-77bdb669d6-5cqbb" podUID="2a5c96aa-dd65-446c-a94d-46676c55163e" containerName="console" containerID="cri-o://962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d" gracePeriod=15 Jul 30 14:50:37.469623 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.469591 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:50:37.488538 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.488187 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dxvgp\" (UniqueName: \"kubernetes.io/projected/2a5c96aa-dd65-446c-a94d-46676c55163e-kube-api-access-dxvgp\") pod \"2a5c96aa-dd65-446c-a94d-46676c55163e\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " Jul 30 14:50:37.488538 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.488237 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-trusted-ca-bundle\") pod \"2a5c96aa-dd65-446c-a94d-46676c55163e\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " Jul 30 14:50:37.488538 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.488315 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-serving-cert\") pod \"2a5c96aa-dd65-446c-a94d-46676c55163e\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " Jul 30 14:50:37.488538 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.488339 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-console-config\") pod \"2a5c96aa-dd65-446c-a94d-46676c55163e\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " Jul 30 14:50:37.488538 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.488377 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-oauth-config\") pod \"2a5c96aa-dd65-446c-a94d-46676c55163e\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " Jul 30 14:50:37.488538 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.488403 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-oauth-serving-cert\") pod \"2a5c96aa-dd65-446c-a94d-46676c55163e\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " Jul 30 14:50:37.488538 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.488442 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-service-ca\") pod \"2a5c96aa-dd65-446c-a94d-46676c55163e\" (UID: \"2a5c96aa-dd65-446c-a94d-46676c55163e\") " Jul 30 14:50:37.489124 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.488949 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-service-ca" (OuterVolumeSpecName: "service-ca") pod "2a5c96aa-dd65-446c-a94d-46676c55163e" (UID: "2a5c96aa-dd65-446c-a94d-46676c55163e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:37.489298 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.489260 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "2a5c96aa-dd65-446c-a94d-46676c55163e" (UID: "2a5c96aa-dd65-446c-a94d-46676c55163e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:37.489510 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.489484 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "2a5c96aa-dd65-446c-a94d-46676c55163e" (UID: "2a5c96aa-dd65-446c-a94d-46676c55163e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:37.489589 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.489491 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-console-config" (OuterVolumeSpecName: "console-config") pod "2a5c96aa-dd65-446c-a94d-46676c55163e" (UID: "2a5c96aa-dd65-446c-a94d-46676c55163e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:50:37.491048 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.490980 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a5c96aa-dd65-446c-a94d-46676c55163e-kube-api-access-dxvgp" (OuterVolumeSpecName: "kube-api-access-dxvgp") pod "2a5c96aa-dd65-446c-a94d-46676c55163e" (UID: "2a5c96aa-dd65-446c-a94d-46676c55163e"). InnerVolumeSpecName "kube-api-access-dxvgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:50:37.492225 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.492187 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "2a5c96aa-dd65-446c-a94d-46676c55163e" (UID: "2a5c96aa-dd65-446c-a94d-46676c55163e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:50:37.492374 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.492350 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "2a5c96aa-dd65-446c-a94d-46676c55163e" (UID: "2a5c96aa-dd65-446c-a94d-46676c55163e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:50:37.589139 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.589107 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-trusted-ca-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:37.589139 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.589134 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-serving-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:37.589139 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.589143 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-console-config\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:37.589368 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.589153 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/2a5c96aa-dd65-446c-a94d-46676c55163e-console-oauth-config\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:37.589368 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.589161 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-oauth-serving-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:37.589368 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.589169 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/2a5c96aa-dd65-446c-a94d-46676c55163e-service-ca\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:37.589368 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:37.589177 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxvgp\" (UniqueName: \"kubernetes.io/projected/2a5c96aa-dd65-446c-a94d-46676c55163e-kube-api-access-dxvgp\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:50:38.286838 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.286802 2564 generic.go:356] "Generic (PLEG): container finished" podID="2a5c96aa-dd65-446c-a94d-46676c55163e" containerID="962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d" exitCode=0 Jul 30 14:50:38.287239 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.286852 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-77bdb669d6-5cqbb" Jul 30 14:50:38.287239 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.286885 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bdb669d6-5cqbb" event={"ID":"2a5c96aa-dd65-446c-a94d-46676c55163e","Type":"ContainerDied","Data":"962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d"} Jul 30 14:50:38.287239 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.286938 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-77bdb669d6-5cqbb" event={"ID":"2a5c96aa-dd65-446c-a94d-46676c55163e","Type":"ContainerDied","Data":"9f177c0d3eaa44a93fd37a86f95a5a5bd8687fa06ee84306ffa4d7936b67faf6"} Jul 30 14:50:38.287239 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.286961 2564 scope.go:117] "RemoveContainer" containerID="962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d" Jul 30 14:50:38.295470 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.295453 2564 scope.go:117] "RemoveContainer" containerID="962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d" Jul 30 14:50:38.295761 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:50:38.295725 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d\": container with ID starting with 962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d not found: ID does not exist" containerID="962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d" Jul 30 14:50:38.295854 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.295767 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d"} err="failed to get container status \"962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d\": rpc error: code = NotFound desc = could not find container \"962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d\": container with ID starting with 962dce6c2fe667da5a4d4d6454d5a8dc59a9f6cf8b620f169016b5842dd7d01d not found: ID does not exist" Jul 30 14:50:38.307880 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.307853 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-77bdb669d6-5cqbb"] Jul 30 14:50:38.310743 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:38.310720 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-77bdb669d6-5cqbb"] Jul 30 14:50:39.489001 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:50:39.488967 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a5c96aa-dd65-446c-a94d-46676c55163e" path="/var/lib/kubelet/pods/2a5c96aa-dd65-446c-a94d-46676c55163e/volumes" Jul 30 14:51:12.260770 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.260727 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6db879668d-bhdwm"] Jul 30 14:51:12.261362 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261209 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4253f459-2c3c-427f-b9da-3449dbd7212f" containerName="registry" Jul 30 14:51:12.261362 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261229 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="4253f459-2c3c-427f-b9da-3449dbd7212f" containerName="registry" Jul 30 14:51:12.261362 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261250 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78076b73-933a-4fc4-a9bf-7e09806f9897" containerName="console" Jul 30 14:51:12.261362 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261260 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="78076b73-933a-4fc4-a9bf-7e09806f9897" containerName="console" Jul 30 14:51:12.261362 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261311 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a5c96aa-dd65-446c-a94d-46676c55163e" containerName="console" Jul 30 14:51:12.261362 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261321 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a5c96aa-dd65-446c-a94d-46676c55163e" containerName="console" Jul 30 14:51:12.261650 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261392 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="78076b73-933a-4fc4-a9bf-7e09806f9897" containerName="console" Jul 30 14:51:12.261650 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261405 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a5c96aa-dd65-446c-a94d-46676c55163e" containerName="console" Jul 30 14:51:12.261650 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.261418 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="4253f459-2c3c-427f-b9da-3449dbd7212f" containerName="registry" Jul 30 14:51:12.266198 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.266178 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.274803 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.274777 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db879668d-bhdwm"] Jul 30 14:51:12.377697 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.377662 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-oauth-config\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.377870 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.377718 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-oauth-serving-cert\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.377870 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.377736 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jzqk\" (UniqueName: \"kubernetes.io/projected/049e518e-d6df-428b-9e39-c92b5bebbab6-kube-api-access-5jzqk\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.377870 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.377822 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-serving-cert\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.377870 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.377857 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-service-ca\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.378000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.377885 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-console-config\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.378000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.377913 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-trusted-ca-bundle\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.478685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.478653 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-oauth-serving-cert\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.478685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.478687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jzqk\" (UniqueName: \"kubernetes.io/projected/049e518e-d6df-428b-9e39-c92b5bebbab6-kube-api-access-5jzqk\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.478944 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.478756 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-serving-cert\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.478944 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.478809 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-service-ca\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.478944 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.478839 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-console-config\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.479116 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.478962 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-trusted-ca-bundle\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.479116 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.479010 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-oauth-config\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.479629 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.479602 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-service-ca\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.479785 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.479700 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-console-config\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.479785 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.479709 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-oauth-serving-cert\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.479785 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.479781 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-trusted-ca-bundle\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.481459 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.481437 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-serving-cert\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.481459 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.481457 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-oauth-config\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.486116 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.486096 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jzqk\" (UniqueName: \"kubernetes.io/projected/049e518e-d6df-428b-9e39-c92b5bebbab6-kube-api-access-5jzqk\") pod \"console-6db879668d-bhdwm\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.577355 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.577264 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:12.709018 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:12.708985 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6db879668d-bhdwm"] Jul 30 14:51:12.712340 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:51:12.712304 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod049e518e_d6df_428b_9e39_c92b5bebbab6.slice/crio-073c8a5bc599bb3bfed2efed31403b0c42634d4097b24029cd3866161c2ad1ea WatchSource:0}: Error finding container 073c8a5bc599bb3bfed2efed31403b0c42634d4097b24029cd3866161c2ad1ea: Status 404 returned error can't find the container with id 073c8a5bc599bb3bfed2efed31403b0c42634d4097b24029cd3866161c2ad1ea Jul 30 14:51:13.426563 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:13.426531 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db879668d-bhdwm" event={"ID":"049e518e-d6df-428b-9e39-c92b5bebbab6","Type":"ContainerStarted","Data":"c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18"} Jul 30 14:51:13.426945 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:13.426579 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db879668d-bhdwm" event={"ID":"049e518e-d6df-428b-9e39-c92b5bebbab6","Type":"ContainerStarted","Data":"073c8a5bc599bb3bfed2efed31403b0c42634d4097b24029cd3866161c2ad1ea"} Jul 30 14:51:13.445600 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:13.445545 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6db879668d-bhdwm" podStartSLOduration=1.445527549 podStartE2EDuration="1.445527549s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:51:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:51:13.443766523 +0000 UTC m=+200.427049851" watchObservedRunningTime="2026-07-30 14:51:13.445527549 +0000 UTC m=+200.428810884" Jul 30 14:51:22.578220 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:22.578179 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:22.578220 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:22.578232 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:22.583059 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:22.583037 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:23.469551 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:23.469524 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:51:23.541387 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:23.540276 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645bdb6d5b-lv9sx"] Jul 30 14:51:48.564115 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.564055 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-645bdb6d5b-lv9sx" podUID="802e3adb-22fb-4935-bc9e-6caccba559a1" containerName="console" containerID="cri-o://e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91" gracePeriod=15 Jul 30 14:51:48.830322 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.830297 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:51:48.891611 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.891575 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-trusted-ca-bundle\") pod \"802e3adb-22fb-4935-bc9e-6caccba559a1\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " Jul 30 14:51:48.891784 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.891676 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-oauth-serving-cert\") pod \"802e3adb-22fb-4935-bc9e-6caccba559a1\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " Jul 30 14:51:48.891784 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.891711 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-serving-cert\") pod \"802e3adb-22fb-4935-bc9e-6caccba559a1\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " Jul 30 14:51:48.891784 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.891727 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-console-config\") pod \"802e3adb-22fb-4935-bc9e-6caccba559a1\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " Jul 30 14:51:48.891784 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.891746 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-service-ca\") pod \"802e3adb-22fb-4935-bc9e-6caccba559a1\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " Jul 30 14:51:48.891784 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.891779 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-oauth-config\") pod \"802e3adb-22fb-4935-bc9e-6caccba559a1\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " Jul 30 14:51:48.892034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.891808 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqfgd\" (UniqueName: \"kubernetes.io/projected/802e3adb-22fb-4935-bc9e-6caccba559a1-kube-api-access-gqfgd\") pod \"802e3adb-22fb-4935-bc9e-6caccba559a1\" (UID: \"802e3adb-22fb-4935-bc9e-6caccba559a1\") " Jul 30 14:51:48.892163 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.891994 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "802e3adb-22fb-4935-bc9e-6caccba559a1" (UID: "802e3adb-22fb-4935-bc9e-6caccba559a1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:51:48.892272 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.892180 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "802e3adb-22fb-4935-bc9e-6caccba559a1" (UID: "802e3adb-22fb-4935-bc9e-6caccba559a1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:51:48.892316 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.892265 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-console-config" (OuterVolumeSpecName: "console-config") pod "802e3adb-22fb-4935-bc9e-6caccba559a1" (UID: "802e3adb-22fb-4935-bc9e-6caccba559a1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:51:48.892372 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.892325 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-service-ca" (OuterVolumeSpecName: "service-ca") pod "802e3adb-22fb-4935-bc9e-6caccba559a1" (UID: "802e3adb-22fb-4935-bc9e-6caccba559a1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:51:48.893793 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.893769 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "802e3adb-22fb-4935-bc9e-6caccba559a1" (UID: "802e3adb-22fb-4935-bc9e-6caccba559a1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:51:48.893917 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.893895 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "802e3adb-22fb-4935-bc9e-6caccba559a1" (UID: "802e3adb-22fb-4935-bc9e-6caccba559a1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:51:48.894013 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.893995 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/802e3adb-22fb-4935-bc9e-6caccba559a1-kube-api-access-gqfgd" (OuterVolumeSpecName: "kube-api-access-gqfgd") pod "802e3adb-22fb-4935-bc9e-6caccba559a1" (UID: "802e3adb-22fb-4935-bc9e-6caccba559a1"). InnerVolumeSpecName "kube-api-access-gqfgd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:51:48.993234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.993202 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-service-ca\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:51:48.993234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.993230 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-oauth-config\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:51:48.993234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.993240 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqfgd\" (UniqueName: \"kubernetes.io/projected/802e3adb-22fb-4935-bc9e-6caccba559a1-kube-api-access-gqfgd\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:51:48.993451 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.993248 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-trusted-ca-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:51:48.993451 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.993257 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-oauth-serving-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:51:48.993451 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.993265 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/802e3adb-22fb-4935-bc9e-6caccba559a1-console-serving-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:51:48.993451 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:48.993297 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/802e3adb-22fb-4935-bc9e-6caccba559a1-console-config\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:51:49.571830 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.571801 2564 generic.go:356] "Generic (PLEG): container finished" podID="802e3adb-22fb-4935-bc9e-6caccba559a1" containerID="e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91" exitCode=0 Jul 30 14:51:49.572243 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.571844 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-645bdb6d5b-lv9sx" Jul 30 14:51:49.572243 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.571884 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645bdb6d5b-lv9sx" event={"ID":"802e3adb-22fb-4935-bc9e-6caccba559a1","Type":"ContainerDied","Data":"e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91"} Jul 30 14:51:49.572243 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.571933 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-645bdb6d5b-lv9sx" event={"ID":"802e3adb-22fb-4935-bc9e-6caccba559a1","Type":"ContainerDied","Data":"eaa5369aa52fb357cc7a45049dbf14586feabb548855a2379028323a22816f5e"} Jul 30 14:51:49.572243 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.571952 2564 scope.go:117] "RemoveContainer" containerID="e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91" Jul 30 14:51:49.582102 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.582064 2564 scope.go:117] "RemoveContainer" containerID="e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91" Jul 30 14:51:49.582363 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:51:49.582337 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91\": container with ID starting with e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91 not found: ID does not exist" containerID="e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91" Jul 30 14:51:49.582446 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.582371 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91"} err="failed to get container status \"e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91\": rpc error: code = NotFound desc = could not find container \"e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91\": container with ID starting with e68fc73365a54ce979e84675b2ebe859458244ebe82b46ee99005e424f3e0b91 not found: ID does not exist" Jul 30 14:51:49.591494 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.591315 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-645bdb6d5b-lv9sx"] Jul 30 14:51:49.596094 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:49.594523 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-645bdb6d5b-lv9sx"] Jul 30 14:51:51.489369 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:51:51.489333 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="802e3adb-22fb-4935-bc9e-6caccba559a1" path="/var/lib/kubelet/pods/802e3adb-22fb-4935-bc9e-6caccba559a1/volumes" Jul 30 14:52:11.313939 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.313901 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn"] Jul 30 14:52:11.314395 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.314245 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="802e3adb-22fb-4935-bc9e-6caccba559a1" containerName="console" Jul 30 14:52:11.314395 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.314258 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="802e3adb-22fb-4935-bc9e-6caccba559a1" containerName="console" Jul 30 14:52:11.314395 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.314336 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="802e3adb-22fb-4935-bc9e-6caccba559a1" containerName="console" Jul 30 14:52:11.318812 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.318795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.320991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.320968 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wqz7b\"" Jul 30 14:52:11.320991 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.320977 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 14:52:11.321505 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.321492 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 14:52:11.326334 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.326039 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn"] Jul 30 14:52:11.489650 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.489623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.489838 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.489699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbc82\" (UniqueName: \"kubernetes.io/projected/f5c920ab-81da-481d-9c6c-4f06bdff3671-kube-api-access-bbc82\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.489838 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.489731 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.590737 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.590635 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbc82\" (UniqueName: \"kubernetes.io/projected/f5c920ab-81da-481d-9c6c-4f06bdff3671-kube-api-access-bbc82\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.590737 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.590687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.590974 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.590773 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.591188 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.591167 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.591260 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.591241 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.599375 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.599342 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbc82\" (UniqueName: \"kubernetes.io/projected/f5c920ab-81da-481d-9c6c-4f06bdff3671-kube-api-access-bbc82\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.629963 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.629928 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:11.761709 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:11.761679 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn"] Jul 30 14:52:11.768129 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:52:11.765015 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5c920ab_81da_481d_9c6c_4f06bdff3671.slice/crio-81b643361749720969d56c69952fa383787b0cf9e91820cc7ebbfced13730572 WatchSource:0}: Error finding container 81b643361749720969d56c69952fa383787b0cf9e91820cc7ebbfced13730572: Status 404 returned error can't find the container with id 81b643361749720969d56c69952fa383787b0cf9e91820cc7ebbfced13730572 Jul 30 14:52:12.664680 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:12.664644 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" event={"ID":"f5c920ab-81da-481d-9c6c-4f06bdff3671","Type":"ContainerStarted","Data":"81b643361749720969d56c69952fa383787b0cf9e91820cc7ebbfced13730572"} Jul 30 14:52:19.697682 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:19.697593 2564 generic.go:356] "Generic (PLEG): container finished" podID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerID="4b1cf9337d98ab1e9c7ded4d20225e3f36ea1df85fca0f2ee915a536832ca3c4" exitCode=0 Jul 30 14:52:19.698149 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:19.697683 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" event={"ID":"f5c920ab-81da-481d-9c6c-4f06bdff3671","Type":"ContainerDied","Data":"4b1cf9337d98ab1e9c7ded4d20225e3f36ea1df85fca0f2ee915a536832ca3c4"} Jul 30 14:52:22.711879 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:22.711842 2564 generic.go:356] "Generic (PLEG): container finished" podID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerID="c8a5227b743fe33fa330de91e72bb1884e146c8f0c43d740e7731bd8515e5ef4" exitCode=0 Jul 30 14:52:22.712384 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:22.711883 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" event={"ID":"f5c920ab-81da-481d-9c6c-4f06bdff3671","Type":"ContainerDied","Data":"c8a5227b743fe33fa330de91e72bb1884e146c8f0c43d740e7731bd8515e5ef4"} Jul 30 14:52:30.749542 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:30.749502 2564 generic.go:356] "Generic (PLEG): container finished" podID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerID="a60069b8b3ab6fbccecf7a33ec535818231ab56e46cfd0ac91702ea16a05b941" exitCode=0 Jul 30 14:52:30.749930 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:30.749584 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" event={"ID":"f5c920ab-81da-481d-9c6c-4f06bdff3671","Type":"ContainerDied","Data":"a60069b8b3ab6fbccecf7a33ec535818231ab56e46cfd0ac91702ea16a05b941"} Jul 30 14:52:31.880124 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:31.880100 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:31.954373 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:31.954283 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-util\") pod \"f5c920ab-81da-481d-9c6c-4f06bdff3671\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " Jul 30 14:52:31.954373 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:31.954356 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bbc82\" (UniqueName: \"kubernetes.io/projected/f5c920ab-81da-481d-9c6c-4f06bdff3671-kube-api-access-bbc82\") pod \"f5c920ab-81da-481d-9c6c-4f06bdff3671\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " Jul 30 14:52:31.954594 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:31.954380 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-bundle\") pod \"f5c920ab-81da-481d-9c6c-4f06bdff3671\" (UID: \"f5c920ab-81da-481d-9c6c-4f06bdff3671\") " Jul 30 14:52:31.955042 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:31.955021 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-bundle" (OuterVolumeSpecName: "bundle") pod "f5c920ab-81da-481d-9c6c-4f06bdff3671" (UID: "f5c920ab-81da-481d-9c6c-4f06bdff3671"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:52:31.956521 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:31.956499 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5c920ab-81da-481d-9c6c-4f06bdff3671-kube-api-access-bbc82" (OuterVolumeSpecName: "kube-api-access-bbc82") pod "f5c920ab-81da-481d-9c6c-4f06bdff3671" (UID: "f5c920ab-81da-481d-9c6c-4f06bdff3671"). InnerVolumeSpecName "kube-api-access-bbc82". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:52:31.959188 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:31.959163 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-util" (OuterVolumeSpecName: "util") pod "f5c920ab-81da-481d-9c6c-4f06bdff3671" (UID: "f5c920ab-81da-481d-9c6c-4f06bdff3671"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:52:32.055480 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:32.055429 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbc82\" (UniqueName: \"kubernetes.io/projected/f5c920ab-81da-481d-9c6c-4f06bdff3671-kube-api-access-bbc82\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:52:32.055480 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:32.055474 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:52:32.055480 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:32.055487 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/f5c920ab-81da-481d-9c6c-4f06bdff3671-util\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:52:32.759655 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:32.759606 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" event={"ID":"f5c920ab-81da-481d-9c6c-4f06bdff3671","Type":"ContainerDied","Data":"81b643361749720969d56c69952fa383787b0cf9e91820cc7ebbfced13730572"} Jul 30 14:52:32.759851 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:32.759671 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81b643361749720969d56c69952fa383787b0cf9e91820cc7ebbfced13730572" Jul 30 14:52:32.759851 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:32.759614 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540mlkxn" Jul 30 14:52:39.356894 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.356859 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x"] Jul 30 14:52:39.357296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.357189 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerName="extract" Jul 30 14:52:39.357296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.357201 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerName="extract" Jul 30 14:52:39.357296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.357211 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerName="pull" Jul 30 14:52:39.357296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.357217 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerName="pull" Jul 30 14:52:39.357296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.357230 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerName="util" Jul 30 14:52:39.357296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.357236 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerName="util" Jul 30 14:52:39.357296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.357291 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5c920ab-81da-481d-9c6c-4f06bdff3671" containerName="extract" Jul 30 14:52:39.400370 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.400331 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x"] Jul 30 14:52:39.400542 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.400433 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" Jul 30 14:52:39.403268 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.403239 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-6wv47\"" Jul 30 14:52:39.403397 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.403251 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 30 14:52:39.403397 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.403381 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 30 14:52:39.411560 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.411536 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dph4x\" (UID: \"ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" Jul 30 14:52:39.411709 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.411597 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7snf\" (UniqueName: \"kubernetes.io/projected/ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3-kube-api-access-m7snf\") pod \"cert-manager-operator-controller-manager-55c444bf86-dph4x\" (UID: \"ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" Jul 30 14:52:39.512375 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.512337 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dph4x\" (UID: \"ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" Jul 30 14:52:39.512557 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.512432 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7snf\" (UniqueName: \"kubernetes.io/projected/ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3-kube-api-access-m7snf\") pod \"cert-manager-operator-controller-manager-55c444bf86-dph4x\" (UID: \"ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" Jul 30 14:52:39.512691 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.512674 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-dph4x\" (UID: \"ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" Jul 30 14:52:39.519918 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.519893 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7snf\" (UniqueName: \"kubernetes.io/projected/ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3-kube-api-access-m7snf\") pod \"cert-manager-operator-controller-manager-55c444bf86-dph4x\" (UID: \"ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" Jul 30 14:52:39.710557 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.710450 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" Jul 30 14:52:39.866209 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:39.866184 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x"] Jul 30 14:52:39.869571 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:52:39.869538 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccdf0f12_b25e_43f7_ae3a_aea6cc15fec3.slice/crio-80d2c24e334ff28d1689ca23187ab3b85dacd1d578ef26e0e4a5c6aa000224f5 WatchSource:0}: Error finding container 80d2c24e334ff28d1689ca23187ab3b85dacd1d578ef26e0e4a5c6aa000224f5: Status 404 returned error can't find the container with id 80d2c24e334ff28d1689ca23187ab3b85dacd1d578ef26e0e4a5c6aa000224f5 Jul 30 14:52:40.793386 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:40.793348 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" event={"ID":"ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3","Type":"ContainerStarted","Data":"80d2c24e334ff28d1689ca23187ab3b85dacd1d578ef26e0e4a5c6aa000224f5"} Jul 30 14:52:42.803237 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:42.803200 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" event={"ID":"ccdf0f12-b25e-43f7-ae3a-aea6cc15fec3","Type":"ContainerStarted","Data":"ea30ad745968a01107ad33acf8fde5906d6159197a28a910d73cb5fcc8914c87"} Jul 30 14:52:42.826324 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:42.826261 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-dph4x" podStartSLOduration=1.745699397 podStartE2EDuration="3.826241247s" totalImagesPullingTime="2.08054185s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:52:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:52:42.825308644 +0000 UTC m=+289.808591979" watchObservedRunningTime="2026-07-30 14:52:42.826241247 +0000 UTC m=+289.809524590" Jul 30 14:52:42.980091 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:42.980041 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kmf2"] Jul 30 14:52:43.005839 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.005808 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kmf2"] Jul 30 14:52:43.005973 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.005907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:43.008397 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.008362 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 30 14:52:43.008397 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.008392 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-8zwps\"" Jul 30 14:52:43.008579 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.008438 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 30 14:52:43.044169 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.044138 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93e86701-28b0-4bf8-b2b2-2a9182c8a8b4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kmf2\" (UID: \"93e86701-28b0-4bf8-b2b2-2a9182c8a8b4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:43.044323 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.044195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cmd4\" (UniqueName: \"kubernetes.io/projected/93e86701-28b0-4bf8-b2b2-2a9182c8a8b4-kube-api-access-6cmd4\") pod \"cert-manager-webhook-6486ffd997-4kmf2\" (UID: \"93e86701-28b0-4bf8-b2b2-2a9182c8a8b4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:43.145545 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.145458 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93e86701-28b0-4bf8-b2b2-2a9182c8a8b4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kmf2\" (UID: \"93e86701-28b0-4bf8-b2b2-2a9182c8a8b4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:43.145545 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.145507 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cmd4\" (UniqueName: \"kubernetes.io/projected/93e86701-28b0-4bf8-b2b2-2a9182c8a8b4-kube-api-access-6cmd4\") pod \"cert-manager-webhook-6486ffd997-4kmf2\" (UID: \"93e86701-28b0-4bf8-b2b2-2a9182c8a8b4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:43.166558 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.166530 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cmd4\" (UniqueName: \"kubernetes.io/projected/93e86701-28b0-4bf8-b2b2-2a9182c8a8b4-kube-api-access-6cmd4\") pod \"cert-manager-webhook-6486ffd997-4kmf2\" (UID: \"93e86701-28b0-4bf8-b2b2-2a9182c8a8b4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:43.172894 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.172868 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/93e86701-28b0-4bf8-b2b2-2a9182c8a8b4-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-4kmf2\" (UID: \"93e86701-28b0-4bf8-b2b2-2a9182c8a8b4\") " pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:43.330175 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.330136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:43.470176 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.470149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-4kmf2"] Jul 30 14:52:43.472648 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:52:43.472617 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93e86701_28b0_4bf8_b2b2_2a9182c8a8b4.slice/crio-27c5ef176e34330faf28026a37a093205b43ff59d9d32c51f9080fd337263cd4 WatchSource:0}: Error finding container 27c5ef176e34330faf28026a37a093205b43ff59d9d32c51f9080fd337263cd4: Status 404 returned error can't find the container with id 27c5ef176e34330faf28026a37a093205b43ff59d9d32c51f9080fd337263cd4 Jul 30 14:52:43.808752 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.808716 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" event={"ID":"93e86701-28b0-4bf8-b2b2-2a9182c8a8b4","Type":"ContainerStarted","Data":"27c5ef176e34330faf28026a37a093205b43ff59d9d32c51f9080fd337263cd4"} Jul 30 14:52:43.811936 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.811905 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn"] Jul 30 14:52:43.814955 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.814936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.817021 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.817005 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 14:52:43.817145 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.817048 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 14:52:43.817145 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.817050 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wqz7b\"" Jul 30 14:52:43.826414 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.826389 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn"] Jul 30 14:52:43.852337 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.852310 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.852485 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.852356 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/605e95c0-7e2b-4663-add7-340660990553-kube-api-access-qgttl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.852485 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.852428 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.953187 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.953147 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/605e95c0-7e2b-4663-add7-340660990553-kube-api-access-qgttl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.953389 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.953228 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.953389 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.953285 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.953640 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.953620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.953711 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.953661 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:43.961839 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:43.961818 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/605e95c0-7e2b-4663-add7-340660990553-kube-api-access-qgttl\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:44.124122 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:44.124020 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:44.282144 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:44.282108 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn"] Jul 30 14:52:44.285045 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:52:44.285015 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605e95c0_7e2b_4663_add7_340660990553.slice/crio-ce862b09d0add61d1fa0ae27f0eaf9de38e1f4d0aec876d0738a2324581982fe WatchSource:0}: Error finding container ce862b09d0add61d1fa0ae27f0eaf9de38e1f4d0aec876d0738a2324581982fe: Status 404 returned error can't find the container with id ce862b09d0add61d1fa0ae27f0eaf9de38e1f4d0aec876d0738a2324581982fe Jul 30 14:52:44.815942 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:44.815899 2564 generic.go:356] "Generic (PLEG): container finished" podID="605e95c0-7e2b-4663-add7-340660990553" containerID="e5e42675bf6476270b8a2634c8d644b16ee0d538a8013eaf895d376e0e4df080" exitCode=0 Jul 30 14:52:44.816478 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:44.816013 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" event={"ID":"605e95c0-7e2b-4663-add7-340660990553","Type":"ContainerDied","Data":"e5e42675bf6476270b8a2634c8d644b16ee0d538a8013eaf895d376e0e4df080"} Jul 30 14:52:44.816478 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:44.816109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" event={"ID":"605e95c0-7e2b-4663-add7-340660990553","Type":"ContainerStarted","Data":"ce862b09d0add61d1fa0ae27f0eaf9de38e1f4d0aec876d0738a2324581982fe"} Jul 30 14:52:46.333739 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.333655 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vskk8"] Jul 30 14:52:46.336057 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.336037 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" Jul 30 14:52:46.337997 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.337979 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-rkqfk\"" Jul 30 14:52:46.346778 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.346752 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vskk8"] Jul 30 14:52:46.375018 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.374985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bb98135-6039-4144-863f-4d6841aea312-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vskk8\" (UID: \"8bb98135-6039-4144-863f-4d6841aea312\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" Jul 30 14:52:46.375154 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.375055 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wgz9\" (UniqueName: \"kubernetes.io/projected/8bb98135-6039-4144-863f-4d6841aea312-kube-api-access-4wgz9\") pod \"cert-manager-cainjector-559cdd8c87-vskk8\" (UID: \"8bb98135-6039-4144-863f-4d6841aea312\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" Jul 30 14:52:46.476111 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.476053 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wgz9\" (UniqueName: \"kubernetes.io/projected/8bb98135-6039-4144-863f-4d6841aea312-kube-api-access-4wgz9\") pod \"cert-manager-cainjector-559cdd8c87-vskk8\" (UID: \"8bb98135-6039-4144-863f-4d6841aea312\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" Jul 30 14:52:46.476288 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.476162 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bb98135-6039-4144-863f-4d6841aea312-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vskk8\" (UID: \"8bb98135-6039-4144-863f-4d6841aea312\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" Jul 30 14:52:46.484575 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.484545 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bb98135-6039-4144-863f-4d6841aea312-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-vskk8\" (UID: \"8bb98135-6039-4144-863f-4d6841aea312\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" Jul 30 14:52:46.484706 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.484657 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wgz9\" (UniqueName: \"kubernetes.io/projected/8bb98135-6039-4144-863f-4d6841aea312-kube-api-access-4wgz9\") pod \"cert-manager-cainjector-559cdd8c87-vskk8\" (UID: \"8bb98135-6039-4144-863f-4d6841aea312\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" Jul 30 14:52:46.645317 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.645235 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" Jul 30 14:52:46.796043 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.796015 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-vskk8"] Jul 30 14:52:46.799263 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:52:46.799231 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8bb98135_6039_4144_863f_4d6841aea312.slice/crio-7252dd64df74db122d45e702c42086bc4c6af8c40b328cf442c613e3087f805f WatchSource:0}: Error finding container 7252dd64df74db122d45e702c42086bc4c6af8c40b328cf442c613e3087f805f: Status 404 returned error can't find the container with id 7252dd64df74db122d45e702c42086bc4c6af8c40b328cf442c613e3087f805f Jul 30 14:52:46.830518 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.830454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" event={"ID":"93e86701-28b0-4bf8-b2b2-2a9182c8a8b4","Type":"ContainerStarted","Data":"c58ce2a1ac574d0a45c76f334962fa6be75d7e83884d735c0ce4dc1b79e75e87"} Jul 30 14:52:46.830659 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.830551 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:46.832149 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.832122 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" event={"ID":"8bb98135-6039-4144-863f-4d6841aea312","Type":"ContainerStarted","Data":"7252dd64df74db122d45e702c42086bc4c6af8c40b328cf442c613e3087f805f"} Jul 30 14:52:46.849162 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:46.849114 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" podStartSLOduration=2.293439948 podStartE2EDuration="4.849054626s" totalImagesPullingTime="2.555614678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:52:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:52:46.848233285 +0000 UTC m=+293.831516623" watchObservedRunningTime="2026-07-30 14:52:46.849054626 +0000 UTC m=+293.832337964" Jul 30 14:52:47.840508 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:47.840470 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" event={"ID":"605e95c0-7e2b-4663-add7-340660990553","Type":"ContainerStarted","Data":"4cd6fc877572ca245da10fd5e1604739e2a2664e63df2428eed52fbc0a88b92a"} Jul 30 14:52:47.842144 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:47.842113 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" event={"ID":"8bb98135-6039-4144-863f-4d6841aea312","Type":"ContainerStarted","Data":"6f412749339aedc30e104fe36e549cbff8aa27df3157f66bd4bc4999035e0ef8"} Jul 30 14:52:47.873988 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:47.873937 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-vskk8" podStartSLOduration=1.873919916 podStartE2EDuration="1.873919916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:52:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:52:47.872727184 +0000 UTC m=+294.856010519" watchObservedRunningTime="2026-07-30 14:52:47.873919916 +0000 UTC m=+294.857203252" Jul 30 14:52:48.849484 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:48.849451 2564 generic.go:356] "Generic (PLEG): container finished" podID="605e95c0-7e2b-4663-add7-340660990553" containerID="4cd6fc877572ca245da10fd5e1604739e2a2664e63df2428eed52fbc0a88b92a" exitCode=0 Jul 30 14:52:48.849858 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:48.849535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" event={"ID":"605e95c0-7e2b-4663-add7-340660990553","Type":"ContainerDied","Data":"4cd6fc877572ca245da10fd5e1604739e2a2664e63df2428eed52fbc0a88b92a"} Jul 30 14:52:49.856060 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:49.856027 2564 generic.go:356] "Generic (PLEG): container finished" podID="605e95c0-7e2b-4663-add7-340660990553" containerID="80ada781878fafc4bc353d9086d1f8187b45635d089d2dda2e088d9626b43b14" exitCode=0 Jul 30 14:52:49.856436 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:49.856117 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" event={"ID":"605e95c0-7e2b-4663-add7-340660990553","Type":"ContainerDied","Data":"80ada781878fafc4bc353d9086d1f8187b45635d089d2dda2e088d9626b43b14"} Jul 30 14:52:50.994360 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:50.994334 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:51.116387 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.116307 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/605e95c0-7e2b-4663-add7-340660990553-kube-api-access-qgttl\") pod \"605e95c0-7e2b-4663-add7-340660990553\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " Jul 30 14:52:51.116530 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.116422 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-util\") pod \"605e95c0-7e2b-4663-add7-340660990553\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " Jul 30 14:52:51.116530 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.116462 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-bundle\") pod \"605e95c0-7e2b-4663-add7-340660990553\" (UID: \"605e95c0-7e2b-4663-add7-340660990553\") " Jul 30 14:52:51.116854 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.116820 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-bundle" (OuterVolumeSpecName: "bundle") pod "605e95c0-7e2b-4663-add7-340660990553" (UID: "605e95c0-7e2b-4663-add7-340660990553"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:52:51.118469 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.118442 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605e95c0-7e2b-4663-add7-340660990553-kube-api-access-qgttl" (OuterVolumeSpecName: "kube-api-access-qgttl") pod "605e95c0-7e2b-4663-add7-340660990553" (UID: "605e95c0-7e2b-4663-add7-340660990553"). InnerVolumeSpecName "kube-api-access-qgttl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:52:51.120842 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.120816 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-util" (OuterVolumeSpecName: "util") pod "605e95c0-7e2b-4663-add7-340660990553" (UID: "605e95c0-7e2b-4663-add7-340660990553"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:52:51.218049 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.218011 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-util\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:52:51.218049 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.218041 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/605e95c0-7e2b-4663-add7-340660990553-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:52:51.218049 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.218050 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgttl\" (UniqueName: \"kubernetes.io/projected/605e95c0-7e2b-4663-add7-340660990553-kube-api-access-qgttl\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:52:51.867016 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.866930 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" Jul 30 14:52:51.867203 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.866925 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87frkkzn" event={"ID":"605e95c0-7e2b-4663-add7-340660990553","Type":"ContainerDied","Data":"ce862b09d0add61d1fa0ae27f0eaf9de38e1f4d0aec876d0738a2324581982fe"} Jul 30 14:52:51.867203 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:51.867051 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce862b09d0add61d1fa0ae27f0eaf9de38e1f4d0aec876d0738a2324581982fe" Jul 30 14:52:52.844350 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:52.844317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-4kmf2" Jul 30 14:52:53.440377 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.440318 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 14:52:53.441573 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.441550 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 14:52:53.447974 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.447953 2564 kubelet.go:1648] "Image garbage collection succeeded" Jul 30 14:52:53.537059 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.537024 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-knpdw"] Jul 30 14:52:53.538271 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.537353 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605e95c0-7e2b-4663-add7-340660990553" containerName="pull" Jul 30 14:52:53.538271 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.537364 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="605e95c0-7e2b-4663-add7-340660990553" containerName="pull" Jul 30 14:52:53.538271 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.537373 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605e95c0-7e2b-4663-add7-340660990553" containerName="util" Jul 30 14:52:53.538271 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.537378 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="605e95c0-7e2b-4663-add7-340660990553" containerName="util" Jul 30 14:52:53.538271 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.537389 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="605e95c0-7e2b-4663-add7-340660990553" containerName="extract" Jul 30 14:52:53.538271 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.537395 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="605e95c0-7e2b-4663-add7-340660990553" containerName="extract" Jul 30 14:52:53.538271 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.537448 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="605e95c0-7e2b-4663-add7-340660990553" containerName="extract" Jul 30 14:52:53.539149 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.539127 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-knpdw" Jul 30 14:52:53.541550 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.541529 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-74p7j\"" Jul 30 14:52:53.551522 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.551493 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-knpdw"] Jul 30 14:52:53.640572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.640527 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kj7hp\" (UniqueName: \"kubernetes.io/projected/c86c45c2-59cc-460c-8a6a-d60abccfeb08-kube-api-access-kj7hp\") pod \"cert-manager-55b4d954d5-knpdw\" (UID: \"c86c45c2-59cc-460c-8a6a-d60abccfeb08\") " pod="cert-manager/cert-manager-55b4d954d5-knpdw" Jul 30 14:52:53.640748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.640628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86c45c2-59cc-460c-8a6a-d60abccfeb08-bound-sa-token\") pod \"cert-manager-55b4d954d5-knpdw\" (UID: \"c86c45c2-59cc-460c-8a6a-d60abccfeb08\") " pod="cert-manager/cert-manager-55b4d954d5-knpdw" Jul 30 14:52:53.741537 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.741505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86c45c2-59cc-460c-8a6a-d60abccfeb08-bound-sa-token\") pod \"cert-manager-55b4d954d5-knpdw\" (UID: \"c86c45c2-59cc-460c-8a6a-d60abccfeb08\") " pod="cert-manager/cert-manager-55b4d954d5-knpdw" Jul 30 14:52:53.741685 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.741557 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kj7hp\" (UniqueName: \"kubernetes.io/projected/c86c45c2-59cc-460c-8a6a-d60abccfeb08-kube-api-access-kj7hp\") pod \"cert-manager-55b4d954d5-knpdw\" (UID: \"c86c45c2-59cc-460c-8a6a-d60abccfeb08\") " pod="cert-manager/cert-manager-55b4d954d5-knpdw" Jul 30 14:52:53.749398 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.749369 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/c86c45c2-59cc-460c-8a6a-d60abccfeb08-bound-sa-token\") pod \"cert-manager-55b4d954d5-knpdw\" (UID: \"c86c45c2-59cc-460c-8a6a-d60abccfeb08\") " pod="cert-manager/cert-manager-55b4d954d5-knpdw" Jul 30 14:52:53.749651 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.749629 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kj7hp\" (UniqueName: \"kubernetes.io/projected/c86c45c2-59cc-460c-8a6a-d60abccfeb08-kube-api-access-kj7hp\") pod \"cert-manager-55b4d954d5-knpdw\" (UID: \"c86c45c2-59cc-460c-8a6a-d60abccfeb08\") " pod="cert-manager/cert-manager-55b4d954d5-knpdw" Jul 30 14:52:53.848597 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.848559 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-knpdw" Jul 30 14:52:53.979895 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:53.979706 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-knpdw"] Jul 30 14:52:53.982416 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:52:53.982387 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc86c45c2_59cc_460c_8a6a_d60abccfeb08.slice/crio-e0c4ed35c75b5a2f75ccc501b768f0c879805b9b4230a082fe8f8dc2b60bd3e4 WatchSource:0}: Error finding container e0c4ed35c75b5a2f75ccc501b768f0c879805b9b4230a082fe8f8dc2b60bd3e4: Status 404 returned error can't find the container with id e0c4ed35c75b5a2f75ccc501b768f0c879805b9b4230a082fe8f8dc2b60bd3e4 Jul 30 14:52:54.881853 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:54.881816 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-knpdw" event={"ID":"c86c45c2-59cc-460c-8a6a-d60abccfeb08","Type":"ContainerStarted","Data":"8359913828f968508886a94d4b3f04652818a09ece881c794f8c30b8bd80d321"} Jul 30 14:52:54.881853 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:54.881859 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-knpdw" event={"ID":"c86c45c2-59cc-460c-8a6a-d60abccfeb08","Type":"ContainerStarted","Data":"e0c4ed35c75b5a2f75ccc501b768f0c879805b9b4230a082fe8f8dc2b60bd3e4"} Jul 30 14:52:54.897838 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:52:54.897784 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-knpdw" podStartSLOduration=1.8977676369999998 podStartE2EDuration="1.897767637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:52:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:52:54.897122054 +0000 UTC m=+301.880405390" watchObservedRunningTime="2026-07-30 14:52:54.897767637 +0000 UTC m=+301.881050972" Jul 30 14:53:10.261587 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.261550 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx"] Jul 30 14:53:10.264290 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.264270 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.266172 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.266149 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 30 14:53:10.266172 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.266166 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 30 14:53:10.266334 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.266240 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-wqz7b\"" Jul 30 14:53:10.273409 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.273384 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx"] Jul 30 14:53:10.393246 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.393211 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.393436 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.393265 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.393436 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.393318 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tlrz\" (UniqueName: \"kubernetes.io/projected/45f64758-97d9-4c73-b06f-892fdbd2b6e5-kube-api-access-5tlrz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.493804 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.493765 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.493980 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.493831 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.493980 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.493865 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tlrz\" (UniqueName: \"kubernetes.io/projected/45f64758-97d9-4c73-b06f-892fdbd2b6e5-kube-api-access-5tlrz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.494234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.494209 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.494297 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.494217 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.501318 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.501285 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tlrz\" (UniqueName: \"kubernetes.io/projected/45f64758-97d9-4c73-b06f-892fdbd2b6e5-kube-api-access-5tlrz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.574465 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.574382 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:10.706069 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.706045 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx"] Jul 30 14:53:10.708660 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:53:10.708632 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45f64758_97d9_4c73_b06f_892fdbd2b6e5.slice/crio-74bc384b6a4d5645390cfa4bb5daaa0d0cda0397c0ece345d4f6cb2f67a76339 WatchSource:0}: Error finding container 74bc384b6a4d5645390cfa4bb5daaa0d0cda0397c0ece345d4f6cb2f67a76339: Status 404 returned error can't find the container with id 74bc384b6a4d5645390cfa4bb5daaa0d0cda0397c0ece345d4f6cb2f67a76339 Jul 30 14:53:10.955870 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.955782 2564 generic.go:356] "Generic (PLEG): container finished" podID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerID="ef3139167f21a204e6942f4b807c6d6c77cc96f262022877f7c710321d7e2b3f" exitCode=0 Jul 30 14:53:10.955870 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.955825 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" event={"ID":"45f64758-97d9-4c73-b06f-892fdbd2b6e5","Type":"ContainerDied","Data":"ef3139167f21a204e6942f4b807c6d6c77cc96f262022877f7c710321d7e2b3f"} Jul 30 14:53:10.955870 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.955869 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" event={"ID":"45f64758-97d9-4c73-b06f-892fdbd2b6e5","Type":"ContainerStarted","Data":"74bc384b6a4d5645390cfa4bb5daaa0d0cda0397c0ece345d4f6cb2f67a76339"} Jul 30 14:53:10.956366 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:10.956349 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 14:53:11.962211 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:11.962129 2564 generic.go:356] "Generic (PLEG): container finished" podID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerID="8562976bcd5fc24bfee7c95fefb49eadeb90342ae5fc03f730e9c430eff781ad" exitCode=0 Jul 30 14:53:11.962211 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:11.962187 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" event={"ID":"45f64758-97d9-4c73-b06f-892fdbd2b6e5","Type":"ContainerDied","Data":"8562976bcd5fc24bfee7c95fefb49eadeb90342ae5fc03f730e9c430eff781ad"} Jul 30 14:53:12.971067 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:12.971028 2564 generic.go:356] "Generic (PLEG): container finished" podID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerID="ac207fe5b28c6479a9cd983324fbbb59977b5978179b7df786f83011d9fc9d75" exitCode=0 Jul 30 14:53:12.971451 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:12.971109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" event={"ID":"45f64758-97d9-4c73-b06f-892fdbd2b6e5","Type":"ContainerDied","Data":"ac207fe5b28c6479a9cd983324fbbb59977b5978179b7df786f83011d9fc9d75"} Jul 30 14:53:14.114856 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.114833 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:14.230830 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.230800 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-util\") pod \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " Jul 30 14:53:14.230992 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.230877 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-bundle\") pod \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " Jul 30 14:53:14.230992 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.230910 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tlrz\" (UniqueName: \"kubernetes.io/projected/45f64758-97d9-4c73-b06f-892fdbd2b6e5-kube-api-access-5tlrz\") pod \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\" (UID: \"45f64758-97d9-4c73-b06f-892fdbd2b6e5\") " Jul 30 14:53:14.231638 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.231604 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-bundle" (OuterVolumeSpecName: "bundle") pod "45f64758-97d9-4c73-b06f-892fdbd2b6e5" (UID: "45f64758-97d9-4c73-b06f-892fdbd2b6e5"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:53:14.232983 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.232961 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45f64758-97d9-4c73-b06f-892fdbd2b6e5-kube-api-access-5tlrz" (OuterVolumeSpecName: "kube-api-access-5tlrz") pod "45f64758-97d9-4c73-b06f-892fdbd2b6e5" (UID: "45f64758-97d9-4c73-b06f-892fdbd2b6e5"). InnerVolumeSpecName "kube-api-access-5tlrz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:53:14.239352 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.239323 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-util" (OuterVolumeSpecName: "util") pod "45f64758-97d9-4c73-b06f-892fdbd2b6e5" (UID: "45f64758-97d9-4c73-b06f-892fdbd2b6e5"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:53:14.331959 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.331920 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-util\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:14.331959 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.331952 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/45f64758-97d9-4c73-b06f-892fdbd2b6e5-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:14.331959 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.331962 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tlrz\" (UniqueName: \"kubernetes.io/projected/45f64758-97d9-4c73-b06f-892fdbd2b6e5-kube-api-access-5tlrz\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:14.982441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.982412 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" Jul 30 14:53:14.982631 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.982411 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afm6glx" event={"ID":"45f64758-97d9-4c73-b06f-892fdbd2b6e5","Type":"ContainerDied","Data":"74bc384b6a4d5645390cfa4bb5daaa0d0cda0397c0ece345d4f6cb2f67a76339"} Jul 30 14:53:14.982631 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:14.982531 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74bc384b6a4d5645390cfa4bb5daaa0d0cda0397c0ece345d4f6cb2f67a76339" Jul 30 14:53:22.098357 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.098325 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh"] Jul 30 14:53:22.098826 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.098673 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerName="util" Jul 30 14:53:22.098826 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.098685 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerName="util" Jul 30 14:53:22.098826 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.098697 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerName="pull" Jul 30 14:53:22.098826 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.098703 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerName="pull" Jul 30 14:53:22.098826 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.098720 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerName="extract" Jul 30 14:53:22.098826 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.098725 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerName="extract" Jul 30 14:53:22.098826 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.098774 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="45f64758-97d9-4c73-b06f-892fdbd2b6e5" containerName="extract" Jul 30 14:53:22.103439 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.103418 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.106499 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.106087 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 30 14:53:22.106499 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.106350 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 30 14:53:22.106731 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.106605 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 30 14:53:22.106906 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.106878 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-zsh4t\"" Jul 30 14:53:22.106997 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.106979 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 30 14:53:22.119362 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.119333 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh"] Jul 30 14:53:22.198318 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.198289 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0570a4c2-9c27-4de2-a4f5-06aac0174584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.198489 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.198329 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98wt8\" (UniqueName: \"kubernetes.io/projected/0570a4c2-9c27-4de2-a4f5-06aac0174584-kube-api-access-98wt8\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.198489 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.198353 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0570a4c2-9c27-4de2-a4f5-06aac0174584-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.198489 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.198454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0570a4c2-9c27-4de2-a4f5-06aac0174584-manifests\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.198623 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.198493 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0570a4c2-9c27-4de2-a4f5-06aac0174584-tmp\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.299760 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.299717 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0570a4c2-9c27-4de2-a4f5-06aac0174584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.299966 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.299766 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98wt8\" (UniqueName: \"kubernetes.io/projected/0570a4c2-9c27-4de2-a4f5-06aac0174584-kube-api-access-98wt8\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.299966 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.299949 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0570a4c2-9c27-4de2-a4f5-06aac0174584-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.300121 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.300049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0570a4c2-9c27-4de2-a4f5-06aac0174584-manifests\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.300208 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.300186 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0570a4c2-9c27-4de2-a4f5-06aac0174584-tmp\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.300411 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.300386 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0570a4c2-9c27-4de2-a4f5-06aac0174584-manifests\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.302108 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.302086 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0570a4c2-9c27-4de2-a4f5-06aac0174584-tmp\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.302441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.302416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0570a4c2-9c27-4de2-a4f5-06aac0174584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.302503 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.302416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0570a4c2-9c27-4de2-a4f5-06aac0174584-webhook-cert\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.309179 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.309159 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98wt8\" (UniqueName: \"kubernetes.io/projected/0570a4c2-9c27-4de2-a4f5-06aac0174584-kube-api-access-98wt8\") pod \"opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh\" (UID: \"0570a4c2-9c27-4de2-a4f5-06aac0174584\") " pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.414549 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.414466 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:22.551542 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:22.551516 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh"] Jul 30 14:53:22.554104 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:53:22.554053 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0570a4c2_9c27_4de2_a4f5_06aac0174584.slice/crio-0deb64ee8fd39be0cdb0c389358abe8e15576a2dd73268584e4b0b73759e5aaa WatchSource:0}: Error finding container 0deb64ee8fd39be0cdb0c389358abe8e15576a2dd73268584e4b0b73759e5aaa: Status 404 returned error can't find the container with id 0deb64ee8fd39be0cdb0c389358abe8e15576a2dd73268584e4b0b73759e5aaa Jul 30 14:53:23.020787 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:23.020750 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" event={"ID":"0570a4c2-9c27-4de2-a4f5-06aac0174584","Type":"ContainerStarted","Data":"0deb64ee8fd39be0cdb0c389358abe8e15576a2dd73268584e4b0b73759e5aaa"} Jul 30 14:53:26.036827 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:26.036784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" event={"ID":"0570a4c2-9c27-4de2-a4f5-06aac0174584","Type":"ContainerStarted","Data":"2acd348b8801e11c3e9653acc6691e42eeec1287b642549ef7b170470a449f55"} Jul 30 14:53:27.042731 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:27.042621 2564 generic.go:356] "Generic (PLEG): container finished" podID="0570a4c2-9c27-4de2-a4f5-06aac0174584" containerID="2acd348b8801e11c3e9653acc6691e42eeec1287b642549ef7b170470a449f55" exitCode=0 Jul 30 14:53:27.043106 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:27.042725 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" event={"ID":"0570a4c2-9c27-4de2-a4f5-06aac0174584","Type":"ContainerDied","Data":"2acd348b8801e11c3e9653acc6691e42eeec1287b642549ef7b170470a449f55"} Jul 30 14:53:28.049157 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:28.049107 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" event={"ID":"0570a4c2-9c27-4de2-a4f5-06aac0174584","Type":"ContainerStarted","Data":"3a89efb5e95e9231e8329f41b7b8581bac1bedad57cbf1a07ba7344894d118d0"} Jul 30 14:53:28.049637 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:28.049215 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:28.076681 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:28.076613 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" podStartSLOduration=1.496264582 podStartE2EDuration="6.07659585s" totalImagesPullingTime="3.490155009s" totalInitContainerRuntime="1.090176259s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:53:22 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:53:28.075233462 +0000 UTC m=+335.058516796" watchObservedRunningTime="2026-07-30 14:53:28.07659585 +0000 UTC m=+335.059879184" Jul 30 14:53:39.056861 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:39.056828 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh" Jul 30 14:53:42.170386 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.170347 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h"] Jul 30 14:53:42.175933 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.175909 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.178250 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.178227 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 30 14:53:42.178489 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.178474 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-h8nw8\"" Jul 30 14:53:42.178882 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.178860 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 30 14:53:42.178982 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.178873 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 30 14:53:42.178982 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.178913 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 30 14:53:42.191016 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.190983 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h"] Jul 30 14:53:42.275544 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.275507 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-tmp\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.275728 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.275576 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkczv\" (UniqueName: \"kubernetes.io/projected/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-kube-api-access-qkczv\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.275728 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.275648 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-tls-certs\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.377003 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.376961 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-tls-certs\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.377207 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.377015 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-tmp\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.377207 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.377093 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkczv\" (UniqueName: \"kubernetes.io/projected/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-kube-api-access-qkczv\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.379486 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.379454 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-tmp\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.379707 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.379688 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-tls-certs\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.388038 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.387995 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkczv\" (UniqueName: \"kubernetes.io/projected/5ceb2d7e-a261-475c-a09d-30a6b77dbe16-kube-api-access-qkczv\") pod \"kube-auth-proxy-59c4bbddff-cfr6h\" (UID: \"5ceb2d7e-a261-475c-a09d-30a6b77dbe16\") " pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.434884 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.434801 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8ncl"] Jul 30 14:53:42.437707 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.437685 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:42.440050 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.440029 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 30 14:53:42.440182 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.440029 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-b7z7r\"" Jul 30 14:53:42.449540 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.449513 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8ncl"] Jul 30 14:53:42.487053 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.487019 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" Jul 30 14:53:42.578883 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.578772 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-cert\") pod \"odh-model-controller-7998c67dc4-p8ncl\" (UID: \"fc4d29fd-f24c-46b9-85cd-7bd355cd9467\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:42.578883 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.578878 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6hgl\" (UniqueName: \"kubernetes.io/projected/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-kube-api-access-d6hgl\") pod \"odh-model-controller-7998c67dc4-p8ncl\" (UID: \"fc4d29fd-f24c-46b9-85cd-7bd355cd9467\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:42.642379 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.642345 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h"] Jul 30 14:53:42.645697 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:53:42.645666 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ceb2d7e_a261_475c_a09d_30a6b77dbe16.slice/crio-fc15c3bca953795237fa94e4851c68e278ad173189cc980210888918cb5a580b WatchSource:0}: Error finding container fc15c3bca953795237fa94e4851c68e278ad173189cc980210888918cb5a580b: Status 404 returned error can't find the container with id fc15c3bca953795237fa94e4851c68e278ad173189cc980210888918cb5a580b Jul 30 14:53:42.680256 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.680222 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-cert\") pod \"odh-model-controller-7998c67dc4-p8ncl\" (UID: \"fc4d29fd-f24c-46b9-85cd-7bd355cd9467\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:42.680430 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.680294 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6hgl\" (UniqueName: \"kubernetes.io/projected/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-kube-api-access-d6hgl\") pod \"odh-model-controller-7998c67dc4-p8ncl\" (UID: \"fc4d29fd-f24c-46b9-85cd-7bd355cd9467\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:42.680430 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:53:42.680393 2564 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 30 14:53:42.680508 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:53:42.680477 2564 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-cert podName:fc4d29fd-f24c-46b9-85cd-7bd355cd9467 nodeName:}" failed. No retries permitted until 2026-07-30 14:53:43.180452492 +0000 UTC m=+350.163735827 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-cert") pod "odh-model-controller-7998c67dc4-p8ncl" (UID: "fc4d29fd-f24c-46b9-85cd-7bd355cd9467") : secret "odh-model-controller-webhook-cert" not found Jul 30 14:53:42.693264 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:42.693201 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6hgl\" (UniqueName: \"kubernetes.io/projected/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-kube-api-access-d6hgl\") pod \"odh-model-controller-7998c67dc4-p8ncl\" (UID: \"fc4d29fd-f24c-46b9-85cd-7bd355cd9467\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:43.124627 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.124594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" event={"ID":"5ceb2d7e-a261-475c-a09d-30a6b77dbe16","Type":"ContainerStarted","Data":"fc15c3bca953795237fa94e4851c68e278ad173189cc980210888918cb5a580b"} Jul 30 14:53:43.185785 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.185738 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-cert\") pod \"odh-model-controller-7998c67dc4-p8ncl\" (UID: \"fc4d29fd-f24c-46b9-85cd-7bd355cd9467\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:43.188168 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.188144 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fc4d29fd-f24c-46b9-85cd-7bd355cd9467-cert\") pod \"odh-model-controller-7998c67dc4-p8ncl\" (UID: \"fc4d29fd-f24c-46b9-85cd-7bd355cd9467\") " pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:43.348705 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.348666 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:43.493548 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.493519 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g"] Jul 30 14:53:43.496517 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.496446 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.498653 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.498598 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 30 14:53:43.498884 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.498864 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 30 14:53:43.499229 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.499169 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 30 14:53:43.500706 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.500685 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-njlc8\"" Jul 30 14:53:43.501045 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.500777 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 30 14:53:43.505656 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.504923 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g"] Jul 30 14:53:43.589420 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.589384 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.589420 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.589419 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x8vt\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-kube-api-access-4x8vt\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.589660 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.589588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.589660 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.589637 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.589762 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.589661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/775313ad-4176-488d-b848-4c694bc4b97e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.589762 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.589699 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.589762 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.589727 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.690597 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.690501 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.690597 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.690558 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.690597 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.690596 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/775313ad-4176-488d-b848-4c694bc4b97e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.690884 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.690650 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.690884 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.690680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.690884 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.690725 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.690884 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.690748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x8vt\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-kube-api-access-4x8vt\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.691846 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.691793 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.693572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.693526 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/775313ad-4176-488d-b848-4c694bc4b97e-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.693572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.693536 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.693964 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.693942 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.694494 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.694470 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.698442 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.698418 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.698875 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.698854 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x8vt\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-kube-api-access-4x8vt\") pod \"istiod-openshift-gateway-7b7995d5cd-qtc2g\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.702441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.702411 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-p8ncl"] Jul 30 14:53:43.706000 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:53:43.705965 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4d29fd_f24c_46b9_85cd_7bd355cd9467.slice/crio-68913ce740d750a28574dc6bd84da6cddb9276b37f0525788ed4b44534d865bd WatchSource:0}: Error finding container 68913ce740d750a28574dc6bd84da6cddb9276b37f0525788ed4b44534d865bd: Status 404 returned error can't find the container with id 68913ce740d750a28574dc6bd84da6cddb9276b37f0525788ed4b44534d865bd Jul 30 14:53:43.816470 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.816182 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:43.975526 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:43.975489 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g"] Jul 30 14:53:43.977471 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:53:43.977431 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775313ad_4176_488d_b848_4c694bc4b97e.slice/crio-2096c4df20aa89de13a26c041871ce1d7f6386282e11d14331b8cf71257f984e WatchSource:0}: Error finding container 2096c4df20aa89de13a26c041871ce1d7f6386282e11d14331b8cf71257f984e: Status 404 returned error can't find the container with id 2096c4df20aa89de13a26c041871ce1d7f6386282e11d14331b8cf71257f984e Jul 30 14:53:44.132330 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:44.132282 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" event={"ID":"fc4d29fd-f24c-46b9-85cd-7bd355cd9467","Type":"ContainerStarted","Data":"68913ce740d750a28574dc6bd84da6cddb9276b37f0525788ed4b44534d865bd"} Jul 30 14:53:44.133726 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:44.133689 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" event={"ID":"775313ad-4176-488d-b848-4c694bc4b97e","Type":"ContainerStarted","Data":"2096c4df20aa89de13a26c041871ce1d7f6386282e11d14331b8cf71257f984e"} Jul 30 14:53:48.432069 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:48.432015 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:53:48.432384 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:48.432117 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:53:49.165561 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.165519 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" event={"ID":"5ceb2d7e-a261-475c-a09d-30a6b77dbe16","Type":"ContainerStarted","Data":"1e676cd87b7c8143b9980be25a9f7a969ec42f6bc7eab766eaef4b190d3401b4"} Jul 30 14:53:49.167365 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.167335 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" event={"ID":"fc4d29fd-f24c-46b9-85cd-7bd355cd9467","Type":"ContainerStarted","Data":"dc31c4665df53d9e7e3b4a2e1a8a8ef525f8fc65deb3e1d6f118a078c125f1a7"} Jul 30 14:53:49.167509 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.167455 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:53:49.169339 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.169308 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" event={"ID":"775313ad-4176-488d-b848-4c694bc4b97e","Type":"ContainerStarted","Data":"6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45"} Jul 30 14:53:49.169458 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.169366 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:49.171417 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.171380 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-qtc2g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 30 14:53:49.171524 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.171448 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" podUID="775313ad-4176-488d-b848-4c694bc4b97e" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 30 14:53:49.184662 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.184606 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-59c4bbddff-cfr6h" podStartSLOduration=3.997733046 podStartE2EDuration="7.184588218s" totalImagesPullingTime="3.186855172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:53:49.184296113 +0000 UTC m=+356.167579448" watchObservedRunningTime="2026-07-30 14:53:49.184588218 +0000 UTC m=+356.167871555" Jul 30 14:53:49.204053 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.204004 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" podStartSLOduration=1.752295972 podStartE2EDuration="6.203987579s" totalImagesPullingTime="4.451691607s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:53:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:53:49.202733446 +0000 UTC m=+356.186016783" watchObservedRunningTime="2026-07-30 14:53:49.203987579 +0000 UTC m=+356.187270911" Jul 30 14:53:49.221583 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.221530 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" podStartSLOduration=2.552143373 podStartE2EDuration="7.22151362s" totalImagesPullingTime="4.669370247s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:53:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:53:49.220398574 +0000 UTC m=+356.203681909" watchObservedRunningTime="2026-07-30 14:53:49.22151362 +0000 UTC m=+356.204796955" Jul 30 14:53:49.668187 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.668156 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt"] Jul 30 14:53:49.672008 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.671987 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.676616 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.676590 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k8h9r\"" Jul 30 14:53:49.708360 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.708324 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt"] Jul 30 14:53:49.755901 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.755855 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.755901 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.755896 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.756196 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.755928 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.756196 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.755985 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.756196 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.756045 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.756196 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.756068 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7dp9\" (UniqueName: \"kubernetes.io/projected/0e73061e-8050-4b5e-b10e-33bd1e2e9454-kube-api-access-s7dp9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.756196 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.756127 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.756387 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.756189 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.756387 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.756254 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.856785 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.856743 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.856785 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.856785 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.856996 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.856819 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.856996 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.856844 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.856996 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.856885 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.856996 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.856913 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7dp9\" (UniqueName: \"kubernetes.io/projected/0e73061e-8050-4b5e-b10e-33bd1e2e9454-kube-api-access-s7dp9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.856996 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.856944 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.856996 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.856974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.857319 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.857026 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.857319 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.857194 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.857319 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.857259 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.857319 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.857316 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.857526 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.857473 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.857920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.857887 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.859673 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.859647 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.859816 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.859801 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.865029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.865003 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e73061e-8050-4b5e-b10e-33bd1e2e9454-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.865165 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.865142 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7dp9\" (UniqueName: \"kubernetes.io/projected/0e73061e-8050-4b5e-b10e-33bd1e2e9454-kube-api-access-s7dp9\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt\" (UID: \"0e73061e-8050-4b5e-b10e-33bd1e2e9454\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:49.982532 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:49.982501 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:50.143247 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:50.143221 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt"] Jul 30 14:53:50.145221 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:53:50.145192 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e73061e_8050_4b5e_b10e_33bd1e2e9454.slice/crio-3a211344dc75dbf61d3b595973edddf270752be2824e0fbf1694c240a6b84251 WatchSource:0}: Error finding container 3a211344dc75dbf61d3b595973edddf270752be2824e0fbf1694c240a6b84251: Status 404 returned error can't find the container with id 3a211344dc75dbf61d3b595973edddf270752be2824e0fbf1694c240a6b84251 Jul 30 14:53:50.177140 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:50.177096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" event={"ID":"0e73061e-8050-4b5e-b10e-33bd1e2e9454","Type":"ContainerStarted","Data":"3a211344dc75dbf61d3b595973edddf270752be2824e0fbf1694c240a6b84251"} Jul 30 14:53:50.177938 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:50.177916 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:53.077196 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:53.077160 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:53:53.077602 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:53.077249 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:53:53.077602 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:53.077300 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:53:53.197926 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:53.197882 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" event={"ID":"0e73061e-8050-4b5e-b10e-33bd1e2e9454","Type":"ContainerStarted","Data":"04c5cc1cece294e4c993cd069644eb74b494626326c944aded64332011a867cf"} Jul 30 14:53:53.244945 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:53.244893 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" podStartSLOduration=1.315316014 podStartE2EDuration="4.244872901s" totalImagesPullingTime="2.929556887s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:53:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:53:53.244189732 +0000 UTC m=+360.227473093" watchObservedRunningTime="2026-07-30 14:53:53.244872901 +0000 UTC m=+360.228156643" Jul 30 14:53:53.983760 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:53.983714 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:54.559095 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.552667 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46w6t"] Jul 30 14:53:54.559095 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.558591 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:54.578451 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.578417 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Jul 30 14:53:54.582428 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.582408 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wkc9s\"" Jul 30 14:53:54.618710 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.618661 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4"] Jul 30 14:53:54.622509 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.622488 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:54.628013 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.627988 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Jul 30 14:53:54.628282 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.628031 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lgv9x\"" Jul 30 14:53:54.647767 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.647730 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46w6t"] Jul 30 14:53:54.690715 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.690683 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4"] Jul 30 14:53:54.707655 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.707623 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fkph\" (UniqueName: \"kubernetes.io/projected/ffe366d3-2a9c-468f-90ce-b998446d83b8-kube-api-access-7fkph\") pod \"llmisvc-controller-manager-777c6bd7d5-4zhf4\" (UID: \"ffe366d3-2a9c-468f-90ce-b998446d83b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:54.707831 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.707678 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5-cert\") pod \"kserve-controller-manager-d6f4b65b8-46w6t\" (UID: \"56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:54.707831 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.707748 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffe366d3-2a9c-468f-90ce-b998446d83b8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4zhf4\" (UID: \"ffe366d3-2a9c-468f-90ce-b998446d83b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:54.707831 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.707804 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thm96\" (UniqueName: \"kubernetes.io/projected/56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5-kube-api-access-thm96\") pod \"kserve-controller-manager-d6f4b65b8-46w6t\" (UID: \"56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:54.809015 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.808983 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fkph\" (UniqueName: \"kubernetes.io/projected/ffe366d3-2a9c-468f-90ce-b998446d83b8-kube-api-access-7fkph\") pod \"llmisvc-controller-manager-777c6bd7d5-4zhf4\" (UID: \"ffe366d3-2a9c-468f-90ce-b998446d83b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:54.809288 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.809033 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5-cert\") pod \"kserve-controller-manager-d6f4b65b8-46w6t\" (UID: \"56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:54.809288 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.809067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffe366d3-2a9c-468f-90ce-b998446d83b8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4zhf4\" (UID: \"ffe366d3-2a9c-468f-90ce-b998446d83b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:54.809288 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.809117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thm96\" (UniqueName: \"kubernetes.io/projected/56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5-kube-api-access-thm96\") pod \"kserve-controller-manager-d6f4b65b8-46w6t\" (UID: \"56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:54.811635 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.811608 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5-cert\") pod \"kserve-controller-manager-d6f4b65b8-46w6t\" (UID: \"56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:54.811735 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.811693 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ffe366d3-2a9c-468f-90ce-b998446d83b8-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-4zhf4\" (UID: \"ffe366d3-2a9c-468f-90ce-b998446d83b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:54.819665 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.819641 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fkph\" (UniqueName: \"kubernetes.io/projected/ffe366d3-2a9c-468f-90ce-b998446d83b8-kube-api-access-7fkph\") pod \"llmisvc-controller-manager-777c6bd7d5-4zhf4\" (UID: \"ffe366d3-2a9c-468f-90ce-b998446d83b8\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:54.819895 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.819875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thm96\" (UniqueName: \"kubernetes.io/projected/56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5-kube-api-access-thm96\") pod \"kserve-controller-manager-d6f4b65b8-46w6t\" (UID: \"56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:54.869990 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.869946 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:54.934471 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.934366 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:54.987456 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:54.987426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:55.030577 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:55.030543 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-46w6t"] Jul 30 14:53:55.032904 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:53:55.032872 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56c5ef2f_c7be_4fa8_a7c5_d4eb631201a5.slice/crio-6f4999b0790db9195efc6bf65204e322675ab52ef17a6bd110871d56869bc044 WatchSource:0}: Error finding container 6f4999b0790db9195efc6bf65204e322675ab52ef17a6bd110871d56869bc044: Status 404 returned error can't find the container with id 6f4999b0790db9195efc6bf65204e322675ab52ef17a6bd110871d56869bc044 Jul 30 14:53:55.102983 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:55.102940 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4"] Jul 30 14:53:55.105146 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:53:55.105121 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podffe366d3_2a9c_468f_90ce_b998446d83b8.slice/crio-58567c6625e062dbf0e3fb7023b71e1d5d14249349216ebd1037748a42fc79dc WatchSource:0}: Error finding container 58567c6625e062dbf0e3fb7023b71e1d5d14249349216ebd1037748a42fc79dc: Status 404 returned error can't find the container with id 58567c6625e062dbf0e3fb7023b71e1d5d14249349216ebd1037748a42fc79dc Jul 30 14:53:55.208406 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:55.208370 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" event={"ID":"56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5","Type":"ContainerStarted","Data":"6f4999b0790db9195efc6bf65204e322675ab52ef17a6bd110871d56869bc044"} Jul 30 14:53:55.209531 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:55.209506 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" event={"ID":"ffe366d3-2a9c-468f-90ce-b998446d83b8","Type":"ContainerStarted","Data":"58567c6625e062dbf0e3fb7023b71e1d5d14249349216ebd1037748a42fc79dc"} Jul 30 14:53:55.209691 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:55.209680 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:55.210549 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:55.210535 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt" Jul 30 14:53:56.106668 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.106588 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g"] Jul 30 14:53:56.107143 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.107113 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" podUID="775313ad-4176-488d-b848-4c694bc4b97e" containerName="discovery" containerID="cri-o://6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45" gracePeriod=30 Jul 30 14:53:56.178168 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.178123 2564 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-qtc2g container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.34:8080/ready\": dial tcp 10.132.0.34:8080: connect: connection refused" start-of-body= Jul 30 14:53:56.178332 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.178203 2564 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" podUID="775313ad-4176-488d-b848-4c694bc4b97e" containerName="discovery" probeResult="failure" output="Get \"http://10.132.0.34:8080/ready\": dial tcp 10.132.0.34:8080: connect: connection refused" Jul 30 14:53:56.406108 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.406060 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:56.523272 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.523233 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-dns-cert\") pod \"775313ad-4176-488d-b848-4c694bc4b97e\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " Jul 30 14:53:56.523456 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.523285 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/775313ad-4176-488d-b848-4c694bc4b97e-local-certs\") pod \"775313ad-4176-488d-b848-4c694bc4b97e\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " Jul 30 14:53:56.523456 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.523320 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-istio-token\") pod \"775313ad-4176-488d-b848-4c694bc4b97e\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " Jul 30 14:53:56.523456 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.523345 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-cacerts\") pod \"775313ad-4176-488d-b848-4c694bc4b97e\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " Jul 30 14:53:56.523456 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.523391 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4x8vt\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-kube-api-access-4x8vt\") pod \"775313ad-4176-488d-b848-4c694bc4b97e\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " Jul 30 14:53:56.523725 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.523473 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-ca-configmap\") pod \"775313ad-4176-488d-b848-4c694bc4b97e\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " Jul 30 14:53:56.523725 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.523581 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-kubeconfig\") pod \"775313ad-4176-488d-b848-4c694bc4b97e\" (UID: \"775313ad-4176-488d-b848-4c694bc4b97e\") " Jul 30 14:53:56.524988 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.524576 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "775313ad-4176-488d-b848-4c694bc4b97e" (UID: "775313ad-4176-488d-b848-4c694bc4b97e"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:53:56.526582 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.526547 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "775313ad-4176-488d-b848-4c694bc4b97e" (UID: "775313ad-4176-488d-b848-4c694bc4b97e"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:53:56.526895 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.526868 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-istio-token" (OuterVolumeSpecName: "istio-token") pod "775313ad-4176-488d-b848-4c694bc4b97e" (UID: "775313ad-4176-488d-b848-4c694bc4b97e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:53:56.527131 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.527061 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "775313ad-4176-488d-b848-4c694bc4b97e" (UID: "775313ad-4176-488d-b848-4c694bc4b97e"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:53:56.527528 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.527493 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-kube-api-access-4x8vt" (OuterVolumeSpecName: "kube-api-access-4x8vt") pod "775313ad-4176-488d-b848-4c694bc4b97e" (UID: "775313ad-4176-488d-b848-4c694bc4b97e"). InnerVolumeSpecName "kube-api-access-4x8vt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:53:56.528441 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.528413 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/775313ad-4176-488d-b848-4c694bc4b97e-local-certs" (OuterVolumeSpecName: "local-certs") pod "775313ad-4176-488d-b848-4c694bc4b97e" (UID: "775313ad-4176-488d-b848-4c694bc4b97e"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:53:56.528751 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.528710 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-cacerts" (OuterVolumeSpecName: "cacerts") pod "775313ad-4176-488d-b848-4c694bc4b97e" (UID: "775313ad-4176-488d-b848-4c694bc4b97e"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:53:56.625226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.625137 2564 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-kubeconfig\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:56.625226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.625166 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-dns-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:56.625226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.625175 2564 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/775313ad-4176-488d-b848-4c694bc4b97e-local-certs\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:56.625226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.625183 2564 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-istio-token\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:56.625226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.625190 2564 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/775313ad-4176-488d-b848-4c694bc4b97e-cacerts\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:56.625226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.625198 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4x8vt\" (UniqueName: \"kubernetes.io/projected/775313ad-4176-488d-b848-4c694bc4b97e-kube-api-access-4x8vt\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:56.625226 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:56.625206 2564 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/775313ad-4176-488d-b848-4c694bc4b97e-istio-csr-ca-configmap\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:53:57.223274 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.223237 2564 generic.go:356] "Generic (PLEG): container finished" podID="775313ad-4176-488d-b848-4c694bc4b97e" containerID="6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45" exitCode=0 Jul 30 14:53:57.223848 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.223302 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" Jul 30 14:53:57.223848 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.223330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" event={"ID":"775313ad-4176-488d-b848-4c694bc4b97e","Type":"ContainerDied","Data":"6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45"} Jul 30 14:53:57.223848 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.223393 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g" event={"ID":"775313ad-4176-488d-b848-4c694bc4b97e","Type":"ContainerDied","Data":"2096c4df20aa89de13a26c041871ce1d7f6386282e11d14331b8cf71257f984e"} Jul 30 14:53:57.223848 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.223416 2564 scope.go:117] "RemoveContainer" containerID="6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45" Jul 30 14:53:57.250684 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.250652 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g"] Jul 30 14:53:57.257582 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.257545 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-qtc2g"] Jul 30 14:53:57.364349 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.364271 2564 scope.go:117] "RemoveContainer" containerID="6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45" Jul 30 14:53:57.364720 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:53:57.364688 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45\": container with ID starting with 6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45 not found: ID does not exist" containerID="6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45" Jul 30 14:53:57.364804 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.364737 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45"} err="failed to get container status \"6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45\": rpc error: code = NotFound desc = could not find container \"6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45\": container with ID starting with 6d3145dcdd0e7885fc2779a0a84bc399f531efd4fb948bd272f959639c1f2c45 not found: ID does not exist" Jul 30 14:53:57.499178 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:57.499141 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="775313ad-4176-488d-b848-4c694bc4b97e" path="/var/lib/kubelet/pods/775313ad-4176-488d-b848-4c694bc4b97e/volumes" Jul 30 14:53:59.238703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:59.238661 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" event={"ID":"56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5","Type":"ContainerStarted","Data":"4065ec5d6ef6e762b33c98447f41ef9931c95c96f1abb8cece383d8e2cb14c49"} Jul 30 14:53:59.239213 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:59.238836 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:53:59.240264 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:59.240237 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" event={"ID":"ffe366d3-2a9c-468f-90ce-b998446d83b8","Type":"ContainerStarted","Data":"d99cee559cd3f9799f4686d58035000ebe1e04c0b003d58f1da6f1966a5dfb8f"} Jul 30 14:53:59.240390 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:59.240343 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:53:59.257123 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:59.257054 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" podStartSLOduration=2.550396536 podStartE2EDuration="5.257039579s" totalImagesPullingTime="2.706643043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:53:59.256140362 +0000 UTC m=+366.239423701" watchObservedRunningTime="2026-07-30 14:53:59.257039579 +0000 UTC m=+366.240322914" Jul 30 14:53:59.275407 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:53:59.275354 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" podStartSLOduration=2.020925316 podStartE2EDuration="5.2753374s" totalImagesPullingTime="3.254412084s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:53:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:53:59.27326082 +0000 UTC m=+366.256544155" watchObservedRunningTime="2026-07-30 14:53:59.2753374 +0000 UTC m=+366.258620765" Jul 30 14:54:00.179574 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:00.179544 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-p8ncl" Jul 30 14:54:30.248163 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:30.248124 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-4zhf4" Jul 30 14:54:30.251095 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:30.251046 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-46w6t" Jul 30 14:54:43.392762 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.392725 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gvq8x"] Jul 30 14:54:43.393164 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.393139 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="775313ad-4176-488d-b848-4c694bc4b97e" containerName="discovery" Jul 30 14:54:43.393164 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.393151 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="775313ad-4176-488d-b848-4c694bc4b97e" containerName="discovery" Jul 30 14:54:43.393235 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.393210 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="775313ad-4176-488d-b848-4c694bc4b97e" containerName="discovery" Jul 30 14:54:43.403929 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.403907 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" Jul 30 14:54:43.407440 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.407226 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 30 14:54:43.407748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.407720 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 30 14:54:43.407857 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.407801 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-np42k\"" Jul 30 14:54:43.410223 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.410193 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gvq8x"] Jul 30 14:54:43.545499 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.545462 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwl7r\" (UniqueName: \"kubernetes.io/projected/96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83-kube-api-access-jwl7r\") pod \"kuadrant-operator-catalog-gvq8x\" (UID: \"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83\") " pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" Jul 30 14:54:43.646042 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.645957 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwl7r\" (UniqueName: \"kubernetes.io/projected/96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83-kube-api-access-jwl7r\") pod \"kuadrant-operator-catalog-gvq8x\" (UID: \"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83\") " pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" Jul 30 14:54:43.654463 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.654431 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwl7r\" (UniqueName: \"kubernetes.io/projected/96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83-kube-api-access-jwl7r\") pod \"kuadrant-operator-catalog-gvq8x\" (UID: \"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83\") " pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" Jul 30 14:54:43.693699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.693669 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gvq8x"] Jul 30 14:54:43.693974 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.693959 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" Jul 30 14:54:43.824310 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.824192 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gvq8x"] Jul 30 14:54:43.827279 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:54:43.827247 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96b5ed64_d48f_4bc3_b9c9_3fd6032b7b83.slice/crio-46e56dd6e71bedf0199e1efa5ae5e27c02523064d42e7997c521c77014ddc15a WatchSource:0}: Error finding container 46e56dd6e71bedf0199e1efa5ae5e27c02523064d42e7997c521c77014ddc15a: Status 404 returned error can't find the container with id 46e56dd6e71bedf0199e1efa5ae5e27c02523064d42e7997c521c77014ddc15a Jul 30 14:54:43.903430 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.903355 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4mcl5"] Jul 30 14:54:43.906306 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.906291 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:43.914698 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:43.914674 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4mcl5"] Jul 30 14:54:44.049688 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:44.049652 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bsh5\" (UniqueName: \"kubernetes.io/projected/086a4804-a211-4335-ad43-a1c1dade9895-kube-api-access-6bsh5\") pod \"kuadrant-operator-catalog-4mcl5\" (UID: \"086a4804-a211-4335-ad43-a1c1dade9895\") " pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:44.150279 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:44.150242 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bsh5\" (UniqueName: \"kubernetes.io/projected/086a4804-a211-4335-ad43-a1c1dade9895-kube-api-access-6bsh5\") pod \"kuadrant-operator-catalog-4mcl5\" (UID: \"086a4804-a211-4335-ad43-a1c1dade9895\") " pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:44.157997 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:44.157931 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bsh5\" (UniqueName: \"kubernetes.io/projected/086a4804-a211-4335-ad43-a1c1dade9895-kube-api-access-6bsh5\") pod \"kuadrant-operator-catalog-4mcl5\" (UID: \"086a4804-a211-4335-ad43-a1c1dade9895\") " pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:44.217405 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:44.217363 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:44.356564 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:44.356537 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4mcl5"] Jul 30 14:54:44.358366 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:54:44.358341 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod086a4804_a211_4335_ad43_a1c1dade9895.slice/crio-d66667f78df394edd95e843508fc337bc4acf560a0d94ec8f97abbddb06c48a0 WatchSource:0}: Error finding container d66667f78df394edd95e843508fc337bc4acf560a0d94ec8f97abbddb06c48a0: Status 404 returned error can't find the container with id d66667f78df394edd95e843508fc337bc4acf560a0d94ec8f97abbddb06c48a0 Jul 30 14:54:44.466199 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:44.466099 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" event={"ID":"086a4804-a211-4335-ad43-a1c1dade9895","Type":"ContainerStarted","Data":"d66667f78df394edd95e843508fc337bc4acf560a0d94ec8f97abbddb06c48a0"} Jul 30 14:54:44.467280 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:44.467255 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" event={"ID":"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83","Type":"ContainerStarted","Data":"46e56dd6e71bedf0199e1efa5ae5e27c02523064d42e7997c521c77014ddc15a"} Jul 30 14:54:46.478930 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.478888 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" event={"ID":"086a4804-a211-4335-ad43-a1c1dade9895","Type":"ContainerStarted","Data":"be61351a29a29e04a95f0d0eb295b9cf31baa40c2a5e5b21c30640d3a2d7fcb2"} Jul 30 14:54:46.480425 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.480399 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" event={"ID":"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83","Type":"ContainerStarted","Data":"fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed"} Jul 30 14:54:46.480535 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.480426 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" podUID="96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83" containerName="registry-server" containerID="cri-o://fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed" gracePeriod=2 Jul 30 14:54:46.494661 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.494461 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" podStartSLOduration=1.925782119 podStartE2EDuration="3.494446382s" totalImagesPullingTime="1.568664263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:54:46.493391992 +0000 UTC m=+413.476675330" watchObservedRunningTime="2026-07-30 14:54:46.494446382 +0000 UTC m=+413.477729717" Jul 30 14:54:46.507804 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.507740 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" podStartSLOduration=1.408826881 podStartE2EDuration="3.507720436s" totalImagesPullingTime="2.098893555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:54:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:54:46.50717123 +0000 UTC m=+413.490454589" watchObservedRunningTime="2026-07-30 14:54:46.507720436 +0000 UTC m=+413.491003773" Jul 30 14:54:46.724904 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.724880 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" Jul 30 14:54:46.876461 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.876377 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwl7r\" (UniqueName: \"kubernetes.io/projected/96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83-kube-api-access-jwl7r\") pod \"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83\" (UID: \"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83\") " Jul 30 14:54:46.878544 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.878515 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83-kube-api-access-jwl7r" (OuterVolumeSpecName: "kube-api-access-jwl7r") pod "96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83" (UID: "96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83"). InnerVolumeSpecName "kube-api-access-jwl7r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:54:46.977458 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:46.977421 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwl7r\" (UniqueName: \"kubernetes.io/projected/96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83-kube-api-access-jwl7r\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:54:47.487125 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.487092 2564 generic.go:356] "Generic (PLEG): container finished" podID="96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83" containerID="fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed" exitCode=0 Jul 30 14:54:47.487512 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.487196 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" Jul 30 14:54:47.489841 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.489809 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" event={"ID":"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83","Type":"ContainerDied","Data":"fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed"} Jul 30 14:54:47.490036 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.489853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-gvq8x" event={"ID":"96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83","Type":"ContainerDied","Data":"46e56dd6e71bedf0199e1efa5ae5e27c02523064d42e7997c521c77014ddc15a"} Jul 30 14:54:47.490036 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.489879 2564 scope.go:117] "RemoveContainer" containerID="fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed" Jul 30 14:54:47.499476 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.499453 2564 scope.go:117] "RemoveContainer" containerID="fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed" Jul 30 14:54:47.499777 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:54:47.499757 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed\": container with ID starting with fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed not found: ID does not exist" containerID="fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed" Jul 30 14:54:47.499843 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.499788 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed"} err="failed to get container status \"fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed\": rpc error: code = NotFound desc = could not find container \"fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed\": container with ID starting with fe677c9f9e1cd3ae5c3103a43f9ce26fbbdd488cf6daeb5036d5240a6314f2ed not found: ID does not exist" Jul 30 14:54:47.510603 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.510563 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gvq8x"] Jul 30 14:54:47.515053 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:47.515031 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-gvq8x"] Jul 30 14:54:49.490182 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:49.490148 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83" path="/var/lib/kubelet/pods/96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83/volumes" Jul 30 14:54:54.218363 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:54.218317 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:54.218803 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:54.218390 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:54.240319 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:54.240286 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:54.543057 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:54.543030 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4mcl5" Jul 30 14:54:58.939581 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.939541 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4"] Jul 30 14:54:58.940159 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.940114 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83" containerName="registry-server" Jul 30 14:54:58.940159 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.940134 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83" containerName="registry-server" Jul 30 14:54:58.940312 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.940297 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="96b5ed64-d48f-4bc3-b9c9-3fd6032b7b83" containerName="registry-server" Jul 30 14:54:58.944747 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.944727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:58.946654 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.946630 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k6jbg\"" Jul 30 14:54:58.952683 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.952651 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4"] Jul 30 14:54:58.981942 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.981912 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:58.982136 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.981958 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:58.982136 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:58.982084 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqfj7\" (UniqueName: \"kubernetes.io/projected/0fb7a89f-57b3-41ab-bdd2-50ce14856956-kube-api-access-rqfj7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:59.083314 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.083275 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:59.083462 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.083353 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqfj7\" (UniqueName: \"kubernetes.io/projected/0fb7a89f-57b3-41ab-bdd2-50ce14856956-kube-api-access-rqfj7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:59.083462 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.083378 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:59.083720 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.083702 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:59.083754 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.083699 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:59.090988 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.090960 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqfj7\" (UniqueName: \"kubernetes.io/projected/0fb7a89f-57b3-41ab-bdd2-50ce14856956-kube-api-access-rqfj7\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:59.254972 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.254936 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:54:59.346788 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.346757 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x"] Jul 30 14:54:59.351859 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.351834 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.361801 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.361772 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x"] Jul 30 14:54:59.385090 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.385040 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5pbb\" (UniqueName: \"kubernetes.io/projected/04c92fc9-14db-40f0-9932-68e69ae8c28c-kube-api-access-t5pbb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.385264 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.385136 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.385264 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.385222 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.401579 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.401553 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4"] Jul 30 14:54:59.404225 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:54:59.404202 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fb7a89f_57b3_41ab_bdd2_50ce14856956.slice/crio-464d5edcf01a58e769471e163b15f3873aa14025329b0426466b18e0eee946a5 WatchSource:0}: Error finding container 464d5edcf01a58e769471e163b15f3873aa14025329b0426466b18e0eee946a5: Status 404 returned error can't find the container with id 464d5edcf01a58e769471e163b15f3873aa14025329b0426466b18e0eee946a5 Jul 30 14:54:59.486000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.485968 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.486163 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.486118 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5pbb\" (UniqueName: \"kubernetes.io/projected/04c92fc9-14db-40f0-9932-68e69ae8c28c-kube-api-access-t5pbb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.486235 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.486187 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.486343 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.486322 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.486476 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.486460 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.498647 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.498617 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5pbb\" (UniqueName: \"kubernetes.io/projected/04c92fc9-14db-40f0-9932-68e69ae8c28c-kube-api-access-t5pbb\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.548177 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.548139 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerID="08affb91eef720a272b97e6340e863de7752f8cae4c145dd556f6257e7172b47" exitCode=0 Jul 30 14:54:59.548341 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.548218 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" event={"ID":"0fb7a89f-57b3-41ab-bdd2-50ce14856956","Type":"ContainerDied","Data":"08affb91eef720a272b97e6340e863de7752f8cae4c145dd556f6257e7172b47"} Jul 30 14:54:59.548341 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.548282 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" event={"ID":"0fb7a89f-57b3-41ab-bdd2-50ce14856956","Type":"ContainerStarted","Data":"464d5edcf01a58e769471e163b15f3873aa14025329b0426466b18e0eee946a5"} Jul 30 14:54:59.664705 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.664665 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:54:59.744114 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.743743 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk"] Jul 30 14:54:59.750609 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.750583 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.757153 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.757121 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk"] Jul 30 14:54:59.788861 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.788780 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.788861 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.788814 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgsl9\" (UniqueName: \"kubernetes.io/projected/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-kube-api-access-rgsl9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.789067 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.788933 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.824996 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.824969 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x"] Jul 30 14:54:59.827001 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:54:59.826971 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c92fc9_14db_40f0_9932_68e69ae8c28c.slice/crio-775ab9ff4150b63c66ff2a9a243ff0fc8d8ee4c08072fabb88c25318fa0cfa81 WatchSource:0}: Error finding container 775ab9ff4150b63c66ff2a9a243ff0fc8d8ee4c08072fabb88c25318fa0cfa81: Status 404 returned error can't find the container with id 775ab9ff4150b63c66ff2a9a243ff0fc8d8ee4c08072fabb88c25318fa0cfa81 Jul 30 14:54:59.889825 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.889799 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.889974 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.889892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.889974 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.889927 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgsl9\" (UniqueName: \"kubernetes.io/projected/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-kube-api-access-rgsl9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.890159 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.890140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.890251 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.890233 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:54:59.898379 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:54:59.898354 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgsl9\" (UniqueName: \"kubernetes.io/projected/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-kube-api-access-rgsl9\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:55:00.060932 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.060835 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:55:00.140525 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.140491 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld"] Jul 30 14:55:00.144626 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.144603 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.155013 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.154218 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld"] Jul 30 14:55:00.192423 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.192392 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.192597 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.192514 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.192597 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.192589 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96xzm\" (UniqueName: \"kubernetes.io/projected/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-kube-api-access-96xzm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.210747 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.210717 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk"] Jul 30 14:55:00.243879 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:00.243844 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1a90ab8_22a9_4ecc_b853_2f587a7ae968.slice/crio-0eac1c70ef40f907a7eb8ba69951b21c7f40c4bbc90fd64d06e4557ee18316ce WatchSource:0}: Error finding container 0eac1c70ef40f907a7eb8ba69951b21c7f40c4bbc90fd64d06e4557ee18316ce: Status 404 returned error can't find the container with id 0eac1c70ef40f907a7eb8ba69951b21c7f40c4bbc90fd64d06e4557ee18316ce Jul 30 14:55:00.292999 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.292974 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.293070 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.293036 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.293124 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.293067 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96xzm\" (UniqueName: \"kubernetes.io/projected/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-kube-api-access-96xzm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.293350 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.293333 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.293408 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.293390 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.301535 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.301513 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96xzm\" (UniqueName: \"kubernetes.io/projected/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-kube-api-access-96xzm\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.458420 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.458391 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:00.556920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.556887 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerID="62d3a11c4155bd20b801d8387a5f2e405fafae61b724336b3d1ff7efbd764200" exitCode=0 Jul 30 14:55:00.557104 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.556973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" event={"ID":"0fb7a89f-57b3-41ab-bdd2-50ce14856956","Type":"ContainerDied","Data":"62d3a11c4155bd20b801d8387a5f2e405fafae61b724336b3d1ff7efbd764200"} Jul 30 14:55:00.558699 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.558674 2564 generic.go:356] "Generic (PLEG): container finished" podID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerID="3676d9c8c3b387f011fa85df2da4ab96e0639580a0aa5d759e5affb8a7a93ee9" exitCode=0 Jul 30 14:55:00.558823 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.558765 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" event={"ID":"04c92fc9-14db-40f0-9932-68e69ae8c28c","Type":"ContainerDied","Data":"3676d9c8c3b387f011fa85df2da4ab96e0639580a0aa5d759e5affb8a7a93ee9"} Jul 30 14:55:00.558823 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.558819 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" event={"ID":"04c92fc9-14db-40f0-9932-68e69ae8c28c","Type":"ContainerStarted","Data":"775ab9ff4150b63c66ff2a9a243ff0fc8d8ee4c08072fabb88c25318fa0cfa81"} Jul 30 14:55:00.561869 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.561825 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerID="4fe664a5612f0e6c5d0a9942077743edf56b6236d42f490ec7394efc20f1f680" exitCode=0 Jul 30 14:55:00.561989 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.561949 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" event={"ID":"d1a90ab8-22a9-4ecc-b853-2f587a7ae968","Type":"ContainerDied","Data":"4fe664a5612f0e6c5d0a9942077743edf56b6236d42f490ec7394efc20f1f680"} Jul 30 14:55:00.562056 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.561989 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" event={"ID":"d1a90ab8-22a9-4ecc-b853-2f587a7ae968","Type":"ContainerStarted","Data":"0eac1c70ef40f907a7eb8ba69951b21c7f40c4bbc90fd64d06e4557ee18316ce"} Jul 30 14:55:00.601175 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:00.601149 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld"] Jul 30 14:55:00.602695 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:00.602671 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c36b98_bfcf_41de_99ef_ad505c4fe8b8.slice/crio-d92060788bec64e8260abcd0b013d536df34137b5b8fdf67964bfd2afc5cacaa WatchSource:0}: Error finding container d92060788bec64e8260abcd0b013d536df34137b5b8fdf67964bfd2afc5cacaa: Status 404 returned error can't find the container with id d92060788bec64e8260abcd0b013d536df34137b5b8fdf67964bfd2afc5cacaa Jul 30 14:55:01.570501 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:01.570461 2564 generic.go:356] "Generic (PLEG): container finished" podID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerID="33c9f4f6b3ea973a3588108b9cfafbf3e784a0d68437d3d99ed7eae4fec6bce9" exitCode=0 Jul 30 14:55:01.570969 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:01.570633 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" event={"ID":"0fb7a89f-57b3-41ab-bdd2-50ce14856956","Type":"ContainerDied","Data":"33c9f4f6b3ea973a3588108b9cfafbf3e784a0d68437d3d99ed7eae4fec6bce9"} Jul 30 14:55:01.574592 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:01.572622 2564 generic.go:356] "Generic (PLEG): container finished" podID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerID="b9083741616d9167ad7390a9189d80dae981e7e8d10879b3c02e0ca9d127a2f3" exitCode=0 Jul 30 14:55:01.574592 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:01.572693 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" event={"ID":"03c36b98-bfcf-41de-99ef-ad505c4fe8b8","Type":"ContainerDied","Data":"b9083741616d9167ad7390a9189d80dae981e7e8d10879b3c02e0ca9d127a2f3"} Jul 30 14:55:01.574592 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:01.572784 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" event={"ID":"03c36b98-bfcf-41de-99ef-ad505c4fe8b8","Type":"ContainerStarted","Data":"d92060788bec64e8260abcd0b013d536df34137b5b8fdf67964bfd2afc5cacaa"} Jul 30 14:55:02.580791 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.580750 2564 generic.go:356] "Generic (PLEG): container finished" podID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerID="7c34dd24244dce0906752b0a33ac3af3cfb4bb3595299a5a108d09b13f898f60" exitCode=0 Jul 30 14:55:02.581269 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.580830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" event={"ID":"04c92fc9-14db-40f0-9932-68e69ae8c28c","Type":"ContainerDied","Data":"7c34dd24244dce0906752b0a33ac3af3cfb4bb3595299a5a108d09b13f898f60"} Jul 30 14:55:02.582814 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.582778 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerID="818d5f619cab5e4c298de487ae7a6815254212717cb184bca853ece3fdc3ca32" exitCode=0 Jul 30 14:55:02.582907 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.582862 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" event={"ID":"d1a90ab8-22a9-4ecc-b853-2f587a7ae968","Type":"ContainerDied","Data":"818d5f619cab5e4c298de487ae7a6815254212717cb184bca853ece3fdc3ca32"} Jul 30 14:55:02.745398 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.745374 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:55:02.816397 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.816369 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-util\") pod \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " Jul 30 14:55:02.816575 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.816435 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqfj7\" (UniqueName: \"kubernetes.io/projected/0fb7a89f-57b3-41ab-bdd2-50ce14856956-kube-api-access-rqfj7\") pod \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " Jul 30 14:55:02.816575 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.816460 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-bundle\") pod \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\" (UID: \"0fb7a89f-57b3-41ab-bdd2-50ce14856956\") " Jul 30 14:55:02.816934 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.816907 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-bundle" (OuterVolumeSpecName: "bundle") pod "0fb7a89f-57b3-41ab-bdd2-50ce14856956" (UID: "0fb7a89f-57b3-41ab-bdd2-50ce14856956"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:02.818407 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.818381 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fb7a89f-57b3-41ab-bdd2-50ce14856956-kube-api-access-rqfj7" (OuterVolumeSpecName: "kube-api-access-rqfj7") pod "0fb7a89f-57b3-41ab-bdd2-50ce14856956" (UID: "0fb7a89f-57b3-41ab-bdd2-50ce14856956"). InnerVolumeSpecName "kube-api-access-rqfj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:55:02.821893 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.821869 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-util" (OuterVolumeSpecName: "util") pod "0fb7a89f-57b3-41ab-bdd2-50ce14856956" (UID: "0fb7a89f-57b3-41ab-bdd2-50ce14856956"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:02.917748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.917710 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-util\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:02.917748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.917739 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqfj7\" (UniqueName: \"kubernetes.io/projected/0fb7a89f-57b3-41ab-bdd2-50ce14856956-kube-api-access-rqfj7\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:02.917748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:02.917749 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/0fb7a89f-57b3-41ab-bdd2-50ce14856956-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:03.590250 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.590219 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" event={"ID":"0fb7a89f-57b3-41ab-bdd2-50ce14856956","Type":"ContainerDied","Data":"464d5edcf01a58e769471e163b15f3873aa14025329b0426466b18e0eee946a5"} Jul 30 14:55:03.590731 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.590251 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4" Jul 30 14:55:03.590731 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.590266 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="464d5edcf01a58e769471e163b15f3873aa14025329b0426466b18e0eee946a5" Jul 30 14:55:03.592892 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.592865 2564 generic.go:356] "Generic (PLEG): container finished" podID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerID="1b66bd4d8c1ae68194a4e782928c5de02d85ef12f54abad842862aab2213dacd" exitCode=0 Jul 30 14:55:03.593036 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.592960 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" event={"ID":"04c92fc9-14db-40f0-9932-68e69ae8c28c","Type":"ContainerDied","Data":"1b66bd4d8c1ae68194a4e782928c5de02d85ef12f54abad842862aab2213dacd"} Jul 30 14:55:03.595250 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.595218 2564 generic.go:356] "Generic (PLEG): container finished" podID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerID="316b5bb9737de157b3c7773404d98255864d797cdaf390c903d36d04fbe6595a" exitCode=0 Jul 30 14:55:03.595377 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.595306 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" event={"ID":"d1a90ab8-22a9-4ecc-b853-2f587a7ae968","Type":"ContainerDied","Data":"316b5bb9737de157b3c7773404d98255864d797cdaf390c903d36d04fbe6595a"} Jul 30 14:55:03.596915 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.596893 2564 generic.go:356] "Generic (PLEG): container finished" podID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerID="346d241adfde7ee2b75e3efc981a0b3b36e26b9937806c1a4ff4d77e41a30cfa" exitCode=0 Jul 30 14:55:03.597014 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:03.596994 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" event={"ID":"03c36b98-bfcf-41de-99ef-ad505c4fe8b8","Type":"ContainerDied","Data":"346d241adfde7ee2b75e3efc981a0b3b36e26b9937806c1a4ff4d77e41a30cfa"} Jul 30 14:55:04.604348 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.604317 2564 generic.go:356] "Generic (PLEG): container finished" podID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerID="fdf1df23bd50eead77655967a8c85d754b866309204adfa3ec96295eadaaa43f" exitCode=0 Jul 30 14:55:04.604747 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.604420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" event={"ID":"03c36b98-bfcf-41de-99ef-ad505c4fe8b8","Type":"ContainerDied","Data":"fdf1df23bd50eead77655967a8c85d754b866309204adfa3ec96295eadaaa43f"} Jul 30 14:55:04.762027 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.761998 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:55:04.788581 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.788557 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:55:04.835349 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.835311 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5pbb\" (UniqueName: \"kubernetes.io/projected/04c92fc9-14db-40f0-9932-68e69ae8c28c-kube-api-access-t5pbb\") pod \"04c92fc9-14db-40f0-9932-68e69ae8c28c\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " Jul 30 14:55:04.835525 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.835364 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-bundle\") pod \"04c92fc9-14db-40f0-9932-68e69ae8c28c\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " Jul 30 14:55:04.835525 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.835392 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-util\") pod \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " Jul 30 14:55:04.835525 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.835427 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-bundle\") pod \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " Jul 30 14:55:04.835525 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.835490 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-util\") pod \"04c92fc9-14db-40f0-9932-68e69ae8c28c\" (UID: \"04c92fc9-14db-40f0-9932-68e69ae8c28c\") " Jul 30 14:55:04.835525 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.835518 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgsl9\" (UniqueName: \"kubernetes.io/projected/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-kube-api-access-rgsl9\") pod \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\" (UID: \"d1a90ab8-22a9-4ecc-b853-2f587a7ae968\") " Jul 30 14:55:04.836156 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.836114 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-bundle" (OuterVolumeSpecName: "bundle") pod "04c92fc9-14db-40f0-9932-68e69ae8c28c" (UID: "04c92fc9-14db-40f0-9932-68e69ae8c28c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:04.836156 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.836125 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-bundle" (OuterVolumeSpecName: "bundle") pod "d1a90ab8-22a9-4ecc-b853-2f587a7ae968" (UID: "d1a90ab8-22a9-4ecc-b853-2f587a7ae968"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:04.837871 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.837844 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04c92fc9-14db-40f0-9932-68e69ae8c28c-kube-api-access-t5pbb" (OuterVolumeSpecName: "kube-api-access-t5pbb") pod "04c92fc9-14db-40f0-9932-68e69ae8c28c" (UID: "04c92fc9-14db-40f0-9932-68e69ae8c28c"). InnerVolumeSpecName "kube-api-access-t5pbb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:55:04.838327 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.838302 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-kube-api-access-rgsl9" (OuterVolumeSpecName: "kube-api-access-rgsl9") pod "d1a90ab8-22a9-4ecc-b853-2f587a7ae968" (UID: "d1a90ab8-22a9-4ecc-b853-2f587a7ae968"). InnerVolumeSpecName "kube-api-access-rgsl9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:55:04.843246 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.843213 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-util" (OuterVolumeSpecName: "util") pod "d1a90ab8-22a9-4ecc-b853-2f587a7ae968" (UID: "d1a90ab8-22a9-4ecc-b853-2f587a7ae968"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:04.843912 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.843885 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-util" (OuterVolumeSpecName: "util") pod "04c92fc9-14db-40f0-9932-68e69ae8c28c" (UID: "04c92fc9-14db-40f0-9932-68e69ae8c28c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:04.937031 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.936949 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-util\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:04.937031 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.936979 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgsl9\" (UniqueName: \"kubernetes.io/projected/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-kube-api-access-rgsl9\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:04.937031 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.936991 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5pbb\" (UniqueName: \"kubernetes.io/projected/04c92fc9-14db-40f0-9932-68e69ae8c28c-kube-api-access-t5pbb\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:04.937031 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.937000 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/04c92fc9-14db-40f0-9932-68e69ae8c28c-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:04.937031 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.937008 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-util\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:04.937031 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:04.937015 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d1a90ab8-22a9-4ecc-b853-2f587a7ae968-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:05.612408 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.612378 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" Jul 30 14:55:05.612820 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.612372 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x" event={"ID":"04c92fc9-14db-40f0-9932-68e69ae8c28c","Type":"ContainerDied","Data":"775ab9ff4150b63c66ff2a9a243ff0fc8d8ee4c08072fabb88c25318fa0cfa81"} Jul 30 14:55:05.612820 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.612505 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="775ab9ff4150b63c66ff2a9a243ff0fc8d8ee4c08072fabb88c25318fa0cfa81" Jul 30 14:55:05.614317 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.614297 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" Jul 30 14:55:05.614317 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.614305 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk" event={"ID":"d1a90ab8-22a9-4ecc-b853-2f587a7ae968","Type":"ContainerDied","Data":"0eac1c70ef40f907a7eb8ba69951b21c7f40c4bbc90fd64d06e4557ee18316ce"} Jul 30 14:55:05.614516 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.614347 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eac1c70ef40f907a7eb8ba69951b21c7f40c4bbc90fd64d06e4557ee18316ce" Jul 30 14:55:05.732786 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.732764 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:05.845824 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.845740 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-bundle\") pod \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " Jul 30 14:55:05.845824 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.845795 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-util\") pod \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " Jul 30 14:55:05.846191 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.845869 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96xzm\" (UniqueName: \"kubernetes.io/projected/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-kube-api-access-96xzm\") pod \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\" (UID: \"03c36b98-bfcf-41de-99ef-ad505c4fe8b8\") " Jul 30 14:55:05.846281 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.846253 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-bundle" (OuterVolumeSpecName: "bundle") pod "03c36b98-bfcf-41de-99ef-ad505c4fe8b8" (UID: "03c36b98-bfcf-41de-99ef-ad505c4fe8b8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:05.847946 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.847923 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-kube-api-access-96xzm" (OuterVolumeSpecName: "kube-api-access-96xzm") pod "03c36b98-bfcf-41de-99ef-ad505c4fe8b8" (UID: "03c36b98-bfcf-41de-99ef-ad505c4fe8b8"). InnerVolumeSpecName "kube-api-access-96xzm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:55:05.851253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.851234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-util" (OuterVolumeSpecName: "util") pod "03c36b98-bfcf-41de-99ef-ad505c4fe8b8" (UID: "03c36b98-bfcf-41de-99ef-ad505c4fe8b8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:05.947419 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.947389 2564 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-util\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:05.947419 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.947413 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96xzm\" (UniqueName: \"kubernetes.io/projected/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-kube-api-access-96xzm\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:05.947419 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:05.947423 2564 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/03c36b98-bfcf-41de-99ef-ad505c4fe8b8-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:06.621368 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:06.621341 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" Jul 30 14:55:06.621727 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:06.621348 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld" event={"ID":"03c36b98-bfcf-41de-99ef-ad505c4fe8b8","Type":"ContainerDied","Data":"d92060788bec64e8260abcd0b013d536df34137b5b8fdf67964bfd2afc5cacaa"} Jul 30 14:55:06.621727 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:06.621451 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92060788bec64e8260abcd0b013d536df34137b5b8fdf67964bfd2afc5cacaa" Jul 30 14:55:13.991577 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.991540 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-86958f67c4-dsw84"] Jul 30 14:55:13.991978 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.991963 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerName="util" Jul 30 14:55:13.992034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.991981 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerName="util" Jul 30 14:55:13.992034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.991997 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerName="pull" Jul 30 14:55:13.992034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992002 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerName="pull" Jul 30 14:55:13.992034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992010 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerName="extract" Jul 30 14:55:13.992034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992015 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerName="extract" Jul 30 14:55:13.992034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992022 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerName="extract" Jul 30 14:55:13.992034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992027 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerName="extract" Jul 30 14:55:13.992034 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992036 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerName="util" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992041 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerName="util" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992047 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerName="extract" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992053 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerName="extract" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992066 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerName="pull" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992086 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerName="pull" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992093 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerName="pull" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992098 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerName="pull" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992105 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerName="pull" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992109 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerName="pull" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992124 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerName="util" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992130 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerName="util" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992142 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerName="extract" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992149 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerName="extract" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992156 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerName="util" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992160 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerName="util" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992223 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fb7a89f-57b3-41ab-bdd2-50ce14856956" containerName="extract" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992233 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="04c92fc9-14db-40f0-9932-68e69ae8c28c" containerName="extract" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992239 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1a90ab8-22a9-4ecc-b853-2f587a7ae968" containerName="extract" Jul 30 14:55:13.992283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.992245 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="03c36b98-bfcf-41de-99ef-ad505c4fe8b8" containerName="extract" Jul 30 14:55:13.995357 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:13.995341 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.005410 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.005383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86958f67c4-dsw84"] Jul 30 14:55:14.123464 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.123421 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-oauth-serving-cert\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.123686 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.123481 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-oauth-config\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.123686 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.123539 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7fkv\" (UniqueName: \"kubernetes.io/projected/ed54d4dd-e381-4203-aa71-5830b90ba9a6-kube-api-access-s7fkv\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.123686 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.123588 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-config\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.123686 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.123611 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-trusted-ca-bundle\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.123686 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.123636 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-service-ca\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.123686 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.123672 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-serving-cert\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.224149 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.224117 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-oauth-config\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.224293 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.224157 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7fkv\" (UniqueName: \"kubernetes.io/projected/ed54d4dd-e381-4203-aa71-5830b90ba9a6-kube-api-access-s7fkv\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.224293 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.224252 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-config\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.224499 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.224421 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-trusted-ca-bundle\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.224499 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.224468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-service-ca\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.224616 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.224526 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-serving-cert\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.224616 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.224601 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-oauth-serving-cert\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.225011 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.224984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-config\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.225283 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.225263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-oauth-serving-cert\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.225371 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.225314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-service-ca\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.225515 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.225490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ed54d4dd-e381-4203-aa71-5830b90ba9a6-trusted-ca-bundle\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.226682 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.226653 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-oauth-config\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.226927 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.226904 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/ed54d4dd-e381-4203-aa71-5830b90ba9a6-console-serving-cert\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.234349 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.234319 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7fkv\" (UniqueName: \"kubernetes.io/projected/ed54d4dd-e381-4203-aa71-5830b90ba9a6-kube-api-access-s7fkv\") pod \"console-86958f67c4-dsw84\" (UID: \"ed54d4dd-e381-4203-aa71-5830b90ba9a6\") " pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.304637 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.304540 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:14.443192 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.443163 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-86958f67c4-dsw84"] Jul 30 14:55:14.445209 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:14.445183 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded54d4dd_e381_4203_aa71_5830b90ba9a6.slice/crio-afe341744c8762b33cde01fafb53b9079197a4de4a23bb0f1a75063672700e52 WatchSource:0}: Error finding container afe341744c8762b33cde01fafb53b9079197a4de4a23bb0f1a75063672700e52: Status 404 returned error can't find the container with id afe341744c8762b33cde01fafb53b9079197a4de4a23bb0f1a75063672700e52 Jul 30 14:55:14.505920 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.505893 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9"] Jul 30 14:55:14.509418 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.509400 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:14.511693 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.511667 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-z5g8m\"" Jul 30 14:55:14.521514 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.521487 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9"] Jul 30 14:55:14.629746 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.629661 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9qb\" (UniqueName: \"kubernetes.io/projected/8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c-kube-api-access-6v9qb\") pod \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" (UID: \"8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:14.666859 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.666811 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86958f67c4-dsw84" event={"ID":"ed54d4dd-e381-4203-aa71-5830b90ba9a6","Type":"ContainerStarted","Data":"78e7bc6435b82560af9d5ca843cfcb1357175b52a3e2ca050dfa624d0fe026bb"} Jul 30 14:55:14.667046 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.666871 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-86958f67c4-dsw84" event={"ID":"ed54d4dd-e381-4203-aa71-5830b90ba9a6","Type":"ContainerStarted","Data":"afe341744c8762b33cde01fafb53b9079197a4de4a23bb0f1a75063672700e52"} Jul 30 14:55:14.693263 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.693202 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-86958f67c4-dsw84" podStartSLOduration=1.6931848600000001 podStartE2EDuration="1.69318486s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:14.691256891 +0000 UTC m=+441.674540284" watchObservedRunningTime="2026-07-30 14:55:14.69318486 +0000 UTC m=+441.676468196" Jul 30 14:55:14.731036 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.730992 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9qb\" (UniqueName: \"kubernetes.io/projected/8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c-kube-api-access-6v9qb\") pod \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" (UID: \"8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:14.741893 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.741866 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9qb\" (UniqueName: \"kubernetes.io/projected/8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c-kube-api-access-6v9qb\") pod \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" (UID: \"8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:14.820869 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.820809 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:14.974284 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:14.974253 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9"] Jul 30 14:55:14.974669 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:14.974644 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa7920b_6a7f_47a0_b8d1_4116d4b44f9c.slice/crio-81bde993dd35e8bee5a9312084a30fa05a9dfd33fdd7eff1c6a4c681502ebb39 WatchSource:0}: Error finding container 81bde993dd35e8bee5a9312084a30fa05a9dfd33fdd7eff1c6a4c681502ebb39: Status 404 returned error can't find the container with id 81bde993dd35e8bee5a9312084a30fa05a9dfd33fdd7eff1c6a4c681502ebb39 Jul 30 14:55:15.673043 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:15.672967 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" event={"ID":"8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c","Type":"ContainerStarted","Data":"81bde993dd35e8bee5a9312084a30fa05a9dfd33fdd7eff1c6a4c681502ebb39"} Jul 30 14:55:17.686878 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:17.686845 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" event={"ID":"8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c","Type":"ContainerStarted","Data":"55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b"} Jul 30 14:55:17.687274 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:17.686980 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:17.705871 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:17.705824 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" podStartSLOduration=1.861309162 podStartE2EDuration="3.705806825s" totalImagesPullingTime="1.844497663s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:17.704340879 +0000 UTC m=+444.687624228" watchObservedRunningTime="2026-07-30 14:55:17.705806825 +0000 UTC m=+444.689090160" Jul 30 14:55:19.887565 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:19.887527 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh"] Jul 30 14:55:19.894172 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:19.894148 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:19.896482 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:19.896459 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4kdz\"" Jul 30 14:55:19.911568 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:19.911536 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh"] Jul 30 14:55:19.981791 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:19.981757 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" (UID: \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:19.981940 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:19.981853 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qsk8\" (UniqueName: \"kubernetes.io/projected/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-kube-api-access-8qsk8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" (UID: \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:20.082406 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:20.082361 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qsk8\" (UniqueName: \"kubernetes.io/projected/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-kube-api-access-8qsk8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" (UID: \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:20.082606 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:20.082467 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" (UID: \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:20.082866 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:20.082843 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" (UID: \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:20.095080 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:20.095054 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qsk8\" (UniqueName: \"kubernetes.io/projected/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-kube-api-access-8qsk8\") pod \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" (UID: \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:20.205334 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:20.205240 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:20.354702 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:20.354676 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh"] Jul 30 14:55:20.356703 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:20.356671 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fc0c698_ec3c_451b_a669_e8e9d50c29cc.slice/crio-80ca19c4fa47279d5699ed093979f752ca331bdd659786b8f27c7ede86ebc0f9 WatchSource:0}: Error finding container 80ca19c4fa47279d5699ed093979f752ca331bdd659786b8f27c7ede86ebc0f9: Status 404 returned error can't find the container with id 80ca19c4fa47279d5699ed093979f752ca331bdd659786b8f27c7ede86ebc0f9 Jul 30 14:55:20.703610 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:20.703578 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" event={"ID":"6fc0c698-ec3c-451b-a669-e8e9d50c29cc","Type":"ContainerStarted","Data":"80ca19c4fa47279d5699ed093979f752ca331bdd659786b8f27c7ede86ebc0f9"} Jul 30 14:55:23.244829 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.244795 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt"] Jul 30 14:55:23.248902 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.248879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" Jul 30 14:55:23.251132 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.251109 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-z5786\"" Jul 30 14:55:23.259203 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.259179 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt"] Jul 30 14:55:23.312420 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.312391 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x68pd\" (UniqueName: \"kubernetes.io/projected/5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd-kube-api-access-x68pd\") pod \"authorino-operator-75cc8d5cb7-qp8pt\" (UID: \"5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" Jul 30 14:55:23.413004 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.412964 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x68pd\" (UniqueName: \"kubernetes.io/projected/5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd-kube-api-access-x68pd\") pod \"authorino-operator-75cc8d5cb7-qp8pt\" (UID: \"5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" Jul 30 14:55:23.421995 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.421961 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x68pd\" (UniqueName: \"kubernetes.io/projected/5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd-kube-api-access-x68pd\") pod \"authorino-operator-75cc8d5cb7-qp8pt\" (UID: \"5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" Jul 30 14:55:23.562125 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.562026 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" Jul 30 14:55:23.715094 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:23.715054 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt"] Jul 30 14:55:23.717414 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:23.717384 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bd7d64b_df24_4133_b7ca_fa0f1ac4a2bd.slice/crio-cc147523a0d5fbd3ab97ce5692cd3569b0d2a376c453ae514acf434d8ed1ea4d WatchSource:0}: Error finding container cc147523a0d5fbd3ab97ce5692cd3569b0d2a376c453ae514acf434d8ed1ea4d: Status 404 returned error can't find the container with id cc147523a0d5fbd3ab97ce5692cd3569b0d2a376c453ae514acf434d8ed1ea4d Jul 30 14:55:24.305185 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:24.305112 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:24.305185 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:24.305177 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:24.311384 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:24.311355 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:24.732014 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:24.731965 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" event={"ID":"5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd","Type":"ContainerStarted","Data":"cc147523a0d5fbd3ab97ce5692cd3569b0d2a376c453ae514acf434d8ed1ea4d"} Jul 30 14:55:24.736577 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:24.736553 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-86958f67c4-dsw84" Jul 30 14:55:24.817442 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:24.817399 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db879668d-bhdwm"] Jul 30 14:55:25.741643 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:25.741605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" event={"ID":"6fc0c698-ec3c-451b-a669-e8e9d50c29cc","Type":"ContainerStarted","Data":"9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e"} Jul 30 14:55:25.742134 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:25.741879 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:25.762053 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:25.761908 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" podStartSLOduration=1.760766058 podStartE2EDuration="6.761889662s" totalImagesPullingTime="5.001123604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:25.760383432 +0000 UTC m=+452.743666770" watchObservedRunningTime="2026-07-30 14:55:25.761889662 +0000 UTC m=+452.745172999" Jul 30 14:55:26.776666 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:26.776628 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m"] Jul 30 14:55:26.788474 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:26.788439 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" Jul 30 14:55:26.790749 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:26.790721 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-9kj87\"" Jul 30 14:55:26.790749 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:26.790731 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 30 14:55:26.792137 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:26.792110 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m"] Jul 30 14:55:26.947874 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:26.947840 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csrlk\" (UniqueName: \"kubernetes.io/projected/9e5b60ac-42ef-4f44-8026-803a4bf045ba-kube-api-access-csrlk\") pod \"dns-operator-controller-manager-864dd78c4c-hcd8m\" (UID: \"9e5b60ac-42ef-4f44-8026-803a4bf045ba\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" Jul 30 14:55:27.048897 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:27.048410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csrlk\" (UniqueName: \"kubernetes.io/projected/9e5b60ac-42ef-4f44-8026-803a4bf045ba-kube-api-access-csrlk\") pod \"dns-operator-controller-manager-864dd78c4c-hcd8m\" (UID: \"9e5b60ac-42ef-4f44-8026-803a4bf045ba\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" Jul 30 14:55:27.063092 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:27.063045 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csrlk\" (UniqueName: \"kubernetes.io/projected/9e5b60ac-42ef-4f44-8026-803a4bf045ba-kube-api-access-csrlk\") pod \"dns-operator-controller-manager-864dd78c4c-hcd8m\" (UID: \"9e5b60ac-42ef-4f44-8026-803a4bf045ba\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" Jul 30 14:55:27.101831 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:27.101792 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" Jul 30 14:55:27.302394 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:27.302357 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m"] Jul 30 14:55:27.302927 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:27.302901 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e5b60ac_42ef_4f44_8026_803a4bf045ba.slice/crio-be5cbaca575fdf52ae14d852c8d665f7612cd6a52ecd02199507c98d037b399b WatchSource:0}: Error finding container be5cbaca575fdf52ae14d852c8d665f7612cd6a52ecd02199507c98d037b399b: Status 404 returned error can't find the container with id be5cbaca575fdf52ae14d852c8d665f7612cd6a52ecd02199507c98d037b399b Jul 30 14:55:27.756488 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:27.756444 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" event={"ID":"5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd","Type":"ContainerStarted","Data":"547337c1cebb1867dc1bb64bc0d5dd8eb501c4ae36ed8a2be7108a2091c007ab"} Jul 30 14:55:27.756684 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:27.756622 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" Jul 30 14:55:27.757794 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:27.757771 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" event={"ID":"9e5b60ac-42ef-4f44-8026-803a4bf045ba","Type":"ContainerStarted","Data":"be5cbaca575fdf52ae14d852c8d665f7612cd6a52ecd02199507c98d037b399b"} Jul 30 14:55:27.777402 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:27.777357 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" podStartSLOduration=1.370755798 podStartE2EDuration="4.777340238s" totalImagesPullingTime="3.40658444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:27.775386561 +0000 UTC m=+454.758669897" watchObservedRunningTime="2026-07-30 14:55:27.777340238 +0000 UTC m=+454.760623574" Jul 30 14:55:28.399729 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.399691 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk"] Jul 30 14:55:28.403296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.403275 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.405422 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.405400 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 30 14:55:28.405632 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.405615 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-k6jbg\"" Jul 30 14:55:28.405632 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.405626 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 30 14:55:28.412191 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.411804 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk"] Jul 30 14:55:28.561261 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.561229 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.561431 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.561283 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5fg\" (UniqueName: \"kubernetes.io/projected/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-kube-api-access-bm5fg\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.561431 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.561378 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.662406 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.662334 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.662406 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.662395 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5fg\" (UniqueName: \"kubernetes.io/projected/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-kube-api-access-bm5fg\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.662606 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.662437 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.663044 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.662994 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.664705 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.664683 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.677677 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.677648 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5fg\" (UniqueName: \"kubernetes.io/projected/e6f8872c-8d81-4c86-9ae5-e33d0935a23d-kube-api-access-bm5fg\") pod \"kuadrant-console-plugin-66b5774985-lw6gk\" (UID: \"e6f8872c-8d81-4c86-9ae5-e33d0935a23d\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:28.694496 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.694466 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:28.714421 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:28.714390 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" Jul 30 14:55:29.157136 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:29.157097 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk"] Jul 30 14:55:29.157994 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:29.157957 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6f8872c_8d81_4c86_9ae5_e33d0935a23d.slice/crio-8e98395eac846022b3c59f218d1f807e7d4ca2d5b609a1b00b8543231588b228 WatchSource:0}: Error finding container 8e98395eac846022b3c59f218d1f807e7d4ca2d5b609a1b00b8543231588b228: Status 404 returned error can't find the container with id 8e98395eac846022b3c59f218d1f807e7d4ca2d5b609a1b00b8543231588b228 Jul 30 14:55:29.772961 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:29.772929 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" event={"ID":"e6f8872c-8d81-4c86-9ae5-e33d0935a23d","Type":"ContainerStarted","Data":"8e98395eac846022b3c59f218d1f807e7d4ca2d5b609a1b00b8543231588b228"} Jul 30 14:55:30.783472 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:30.783323 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" event={"ID":"9e5b60ac-42ef-4f44-8026-803a4bf045ba","Type":"ContainerStarted","Data":"17201174ebfae112a2be89f8fd03a830773a4f5d5d29bc84b0420d7d862bf065"} Jul 30 14:55:30.783472 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:30.783426 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" Jul 30 14:55:30.806091 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:30.806013 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" podStartSLOduration=2.395413815 podStartE2EDuration="4.805992683s" totalImagesPullingTime="2.410578868s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:30.803469693 +0000 UTC m=+457.786753031" watchObservedRunningTime="2026-07-30 14:55:30.805992683 +0000 UTC m=+457.789276022" Jul 30 14:55:36.751473 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:36.751438 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:38.552939 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.552898 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb"] Jul 30 14:55:38.558761 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.558731 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.561785 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.561754 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5013eef1-9b58-46f2-8a95-3706a6e882e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" (UID: \"5013eef1-9b58-46f2-8a95-3706a6e882e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.561907 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.561866 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96tj6\" (UniqueName: \"kubernetes.io/projected/5013eef1-9b58-46f2-8a95-3706a6e882e6-kube-api-access-96tj6\") pod \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" (UID: \"5013eef1-9b58-46f2-8a95-3706a6e882e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.574654 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.574602 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb"] Jul 30 14:55:38.654705 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.654671 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh"] Jul 30 14:55:38.654936 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.654909 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" containerName="manager" containerID="cri-o://9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e" gracePeriod=2 Jul 30 14:55:38.661450 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.661412 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh"] Jul 30 14:55:38.663267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.663237 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5013eef1-9b58-46f2-8a95-3706a6e882e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" (UID: \"5013eef1-9b58-46f2-8a95-3706a6e882e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.663672 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.663627 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5013eef1-9b58-46f2-8a95-3706a6e882e6-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" (UID: \"5013eef1-9b58-46f2-8a95-3706a6e882e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.663937 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.663914 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96tj6\" (UniqueName: \"kubernetes.io/projected/5013eef1-9b58-46f2-8a95-3706a6e882e6-kube-api-access-96tj6\") pod \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" (UID: \"5013eef1-9b58-46f2-8a95-3706a6e882e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.675687 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.675280 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb"] Jul 30 14:55:38.675687 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.675633 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-96tj6], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" podUID="5013eef1-9b58-46f2-8a95-3706a6e882e6" Jul 30 14:55:38.679818 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.679776 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-96tj6\" (UniqueName: \"kubernetes.io/projected/5013eef1-9b58-46f2-8a95-3706a6e882e6-kube-api-access-96tj6\") pod \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" (UID: \"5013eef1-9b58-46f2-8a95-3706a6e882e6\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.683355 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.683323 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8"] Jul 30 14:55:38.683985 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.683963 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" containerName="manager" Jul 30 14:55:38.683985 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.683987 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" containerName="manager" Jul 30 14:55:38.684173 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.684131 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" containerName="manager" Jul 30 14:55:38.687810 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.687787 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb"] Jul 30 14:55:38.687903 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.687871 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:38.692109 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.691596 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:38.697140 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.697111 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9"] Jul 30 14:55:38.697473 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.697305 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" containerName="manager" containerID="cri-o://55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b" gracePeriod=2 Jul 30 14:55:38.705593 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.705562 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8"] Jul 30 14:55:38.709903 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.709872 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg"] Jul 30 14:55:38.714444 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.714420 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9"] Jul 30 14:55:38.714666 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.714528 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:38.730399 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.729726 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg"] Jul 30 14:55:38.737621 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.737583 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s"] Jul 30 14:55:38.738446 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.738404 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" containerName="manager" Jul 30 14:55:38.738446 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.738433 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" containerName="manager" Jul 30 14:55:38.738619 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.738601 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" containerName="manager" Jul 30 14:55:38.742223 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.742201 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" Jul 30 14:55:38.749438 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.749392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s"] Jul 30 14:55:38.751628 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.751594 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:38.753116 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.753090 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:38.754815 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.754785 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:38.765042 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.765006 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsz89\" (UniqueName: \"kubernetes.io/projected/3a86e7cb-c0ab-4858-9507-577203b2cbeb-kube-api-access-vsz89\") pod \"limitador-operator-controller-manager-859d7b78bd-4vg2s\" (UID: \"3a86e7cb-c0ab-4858-9507-577203b2cbeb\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" Jul 30 14:55:38.765208 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.765058 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvp4v\" (UniqueName: \"kubernetes.io/projected/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-kube-api-access-gvp4v\") pod \"kuadrant-operator-controller-manager-86878c769f-4cbsg\" (UID: \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:38.765292 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.765210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d72a343e-ad27-471f-bf13-946695e4396e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nckh8\" (UID: \"d72a343e-ad27-471f-bf13-946695e4396e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:38.765374 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.765351 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4cbsg\" (UID: \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:38.765437 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.765382 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j96kf\" (UniqueName: \"kubernetes.io/projected/d72a343e-ad27-471f-bf13-946695e4396e-kube-api-access-j96kf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nckh8\" (UID: \"d72a343e-ad27-471f-bf13-946695e4396e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:38.768710 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.768679 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-qp8pt" Jul 30 14:55:38.774218 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.774189 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:38.775696 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.775665 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:38.777454 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.777337 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:38.839515 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.839435 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.841481 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.841345 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="5013eef1-9b58-46f2-8a95-3706a6e882e6" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.842873 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.842845 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:38.844498 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.844277 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:38.849167 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.849142 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.851029 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.850992 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:38.852896 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.852859 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:38.854461 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:38.854430 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="5013eef1-9b58-46f2-8a95-3706a6e882e6" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:38.866182 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866130 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5013eef1-9b58-46f2-8a95-3706a6e882e6-extensions-socket-volume\") pod \"5013eef1-9b58-46f2-8a95-3706a6e882e6\" (UID: \"5013eef1-9b58-46f2-8a95-3706a6e882e6\") " Jul 30 14:55:38.866182 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866173 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-96tj6\" (UniqueName: \"kubernetes.io/projected/5013eef1-9b58-46f2-8a95-3706a6e882e6-kube-api-access-96tj6\") pod \"5013eef1-9b58-46f2-8a95-3706a6e882e6\" (UID: \"5013eef1-9b58-46f2-8a95-3706a6e882e6\") " Jul 30 14:55:38.866415 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866393 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5013eef1-9b58-46f2-8a95-3706a6e882e6-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5013eef1-9b58-46f2-8a95-3706a6e882e6" (UID: "5013eef1-9b58-46f2-8a95-3706a6e882e6"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:38.866491 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866400 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsz89\" (UniqueName: \"kubernetes.io/projected/3a86e7cb-c0ab-4858-9507-577203b2cbeb-kube-api-access-vsz89\") pod \"limitador-operator-controller-manager-859d7b78bd-4vg2s\" (UID: \"3a86e7cb-c0ab-4858-9507-577203b2cbeb\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" Jul 30 14:55:38.866491 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866468 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvp4v\" (UniqueName: \"kubernetes.io/projected/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-kube-api-access-gvp4v\") pod \"kuadrant-operator-controller-manager-86878c769f-4cbsg\" (UID: \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:38.866621 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866497 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d72a343e-ad27-471f-bf13-946695e4396e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nckh8\" (UID: \"d72a343e-ad27-471f-bf13-946695e4396e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:38.866687 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866652 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4cbsg\" (UID: \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:38.866687 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866679 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j96kf\" (UniqueName: \"kubernetes.io/projected/d72a343e-ad27-471f-bf13-946695e4396e-kube-api-access-j96kf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nckh8\" (UID: \"d72a343e-ad27-471f-bf13-946695e4396e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:38.866838 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866773 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5013eef1-9b58-46f2-8a95-3706a6e882e6-extensions-socket-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:38.867000 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.866974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d72a343e-ad27-471f-bf13-946695e4396e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nckh8\" (UID: \"d72a343e-ad27-471f-bf13-946695e4396e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:38.867151 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.867033 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-4cbsg\" (UID: \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:38.868672 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.868647 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5013eef1-9b58-46f2-8a95-3706a6e882e6-kube-api-access-96tj6" (OuterVolumeSpecName: "kube-api-access-96tj6") pod "5013eef1-9b58-46f2-8a95-3706a6e882e6" (UID: "5013eef1-9b58-46f2-8a95-3706a6e882e6"). InnerVolumeSpecName "kube-api-access-96tj6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:55:38.875261 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.875202 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsz89\" (UniqueName: \"kubernetes.io/projected/3a86e7cb-c0ab-4858-9507-577203b2cbeb-kube-api-access-vsz89\") pod \"limitador-operator-controller-manager-859d7b78bd-4vg2s\" (UID: \"3a86e7cb-c0ab-4858-9507-577203b2cbeb\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" Jul 30 14:55:38.876058 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.876038 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j96kf\" (UniqueName: \"kubernetes.io/projected/d72a343e-ad27-471f-bf13-946695e4396e-kube-api-access-j96kf\") pod \"kuadrant-operator-controller-manager-8684c9ff49-nckh8\" (UID: \"d72a343e-ad27-471f-bf13-946695e4396e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:38.876185 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.876039 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvp4v\" (UniqueName: \"kubernetes.io/projected/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-kube-api-access-gvp4v\") pod \"kuadrant-operator-controller-manager-86878c769f-4cbsg\" (UID: \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:38.967629 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:38.967589 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96tj6\" (UniqueName: \"kubernetes.io/projected/5013eef1-9b58-46f2-8a95-3706a6e882e6-kube-api-access-96tj6\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:39.086131 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:39.086089 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:39.094148 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:39.094117 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:39.103199 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:39.103145 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" Jul 30 14:55:39.493505 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:39.493469 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5013eef1-9b58-46f2-8a95-3706a6e882e6" path="/var/lib/kubelet/pods/5013eef1-9b58-46f2-8a95-3706a6e882e6/volumes" Jul 30 14:55:39.846287 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:39.846207 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:39.850930 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:39.850893 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-86878c769f-rx6rb\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="5013eef1-9b58-46f2-8a95-3706a6e882e6" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rx6rb" Jul 30 14:55:41.791885 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:41.791852 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-hcd8m" Jul 30 14:55:49.841174 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:49.841099 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6db879668d-bhdwm" podUID="049e518e-d6df-428b-9e39-c92b5bebbab6" containerName="console" containerID="cri-o://c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18" gracePeriod=15 Jul 30 14:55:53.420606 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.420540 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:55:53.425969 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.425944 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:53.428117 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.428062 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:53.448059 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.447916 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:53.449682 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.449643 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:53.497015 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.496963 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:53.498857 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.498823 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:53.515824 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.515803 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-oauth-config\") pod \"049e518e-d6df-428b-9e39-c92b5bebbab6\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " Jul 30 14:55:53.515948 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.515850 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-serving-cert\") pod \"049e518e-d6df-428b-9e39-c92b5bebbab6\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " Jul 30 14:55:53.515948 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.515871 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-trusted-ca-bundle\") pod \"049e518e-d6df-428b-9e39-c92b5bebbab6\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " Jul 30 14:55:53.516063 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516037 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-console-config\") pod \"049e518e-d6df-428b-9e39-c92b5bebbab6\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " Jul 30 14:55:53.516141 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516101 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-oauth-serving-cert\") pod \"049e518e-d6df-428b-9e39-c92b5bebbab6\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " Jul 30 14:55:53.516204 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516174 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-service-ca\") pod \"049e518e-d6df-428b-9e39-c92b5bebbab6\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " Jul 30 14:55:53.516263 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516227 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jzqk\" (UniqueName: \"kubernetes.io/projected/049e518e-d6df-428b-9e39-c92b5bebbab6-kube-api-access-5jzqk\") pod \"049e518e-d6df-428b-9e39-c92b5bebbab6\" (UID: \"049e518e-d6df-428b-9e39-c92b5bebbab6\") " Jul 30 14:55:53.516325 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516259 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "049e518e-d6df-428b-9e39-c92b5bebbab6" (UID: "049e518e-d6df-428b-9e39-c92b5bebbab6"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:55:53.516382 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516289 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-extensions-socket-volume\") pod \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\" (UID: \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\") " Jul 30 14:55:53.516433 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516385 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qsk8\" (UniqueName: \"kubernetes.io/projected/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-kube-api-access-8qsk8\") pod \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\" (UID: \"6fc0c698-ec3c-451b-a669-e8e9d50c29cc\") " Jul 30 14:55:53.516433 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516412 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6v9qb\" (UniqueName: \"kubernetes.io/projected/8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c-kube-api-access-6v9qb\") pod \"8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c\" (UID: \"8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c\") " Jul 30 14:55:53.516570 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516537 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "049e518e-d6df-428b-9e39-c92b5bebbab6" (UID: "049e518e-d6df-428b-9e39-c92b5bebbab6"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:55:53.516621 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516567 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-console-config" (OuterVolumeSpecName: "console-config") pod "049e518e-d6df-428b-9e39-c92b5bebbab6" (UID: "049e518e-d6df-428b-9e39-c92b5bebbab6"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:55:53.516748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516732 2564 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-trusted-ca-bundle\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.516812 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516754 2564 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-console-config\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.516812 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516768 2564 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-oauth-serving-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.516812 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516773 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6fc0c698-ec3c-451b-a669-e8e9d50c29cc" (UID: "6fc0c698-ec3c-451b-a669-e8e9d50c29cc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:55:53.516970 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.516833 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-service-ca" (OuterVolumeSpecName: "service-ca") pod "049e518e-d6df-428b-9e39-c92b5bebbab6" (UID: "049e518e-d6df-428b-9e39-c92b5bebbab6"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 14:55:53.517999 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.517972 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "049e518e-d6df-428b-9e39-c92b5bebbab6" (UID: "049e518e-d6df-428b-9e39-c92b5bebbab6"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:55:53.518171 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.518144 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "049e518e-d6df-428b-9e39-c92b5bebbab6" (UID: "049e518e-d6df-428b-9e39-c92b5bebbab6"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:55:53.518454 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.518429 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-kube-api-access-8qsk8" (OuterVolumeSpecName: "kube-api-access-8qsk8") pod "6fc0c698-ec3c-451b-a669-e8e9d50c29cc" (UID: "6fc0c698-ec3c-451b-a669-e8e9d50c29cc"). InnerVolumeSpecName "kube-api-access-8qsk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:55:53.518539 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.518517 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/049e518e-d6df-428b-9e39-c92b5bebbab6-kube-api-access-5jzqk" (OuterVolumeSpecName: "kube-api-access-5jzqk") pod "049e518e-d6df-428b-9e39-c92b5bebbab6" (UID: "049e518e-d6df-428b-9e39-c92b5bebbab6"). InnerVolumeSpecName "kube-api-access-5jzqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:55:53.518636 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.518616 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c-kube-api-access-6v9qb" (OuterVolumeSpecName: "kube-api-access-6v9qb") pod "8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" (UID: "8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c"). InnerVolumeSpecName "kube-api-access-6v9qb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:55:53.616703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.616546 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s"] Jul 30 14:55:53.616703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.617703 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jzqk\" (UniqueName: \"kubernetes.io/projected/049e518e-d6df-428b-9e39-c92b5bebbab6-kube-api-access-5jzqk\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.616703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.617729 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-extensions-socket-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.616703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.617742 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qsk8\" (UniqueName: \"kubernetes.io/projected/6fc0c698-ec3c-451b-a669-e8e9d50c29cc-kube-api-access-8qsk8\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.616703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.617754 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9qb\" (UniqueName: \"kubernetes.io/projected/8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c-kube-api-access-6v9qb\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.616703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.617767 2564 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-oauth-config\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.616703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.617779 2564 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/049e518e-d6df-428b-9e39-c92b5bebbab6-console-serving-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.616703 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.617791 2564 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/049e518e-d6df-428b-9e39-c92b5bebbab6-service-ca\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:55:53.619359 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:53.619322 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a86e7cb_c0ab_4858_9507_577203b2cbeb.slice/crio-3cd8defe9d6b1e48556e830687ba752985cf41357f6b22cfeee98a4e5da101f5 WatchSource:0}: Error finding container 3cd8defe9d6b1e48556e830687ba752985cf41357f6b22cfeee98a4e5da101f5: Status 404 returned error can't find the container with id 3cd8defe9d6b1e48556e830687ba752985cf41357f6b22cfeee98a4e5da101f5 Jul 30 14:55:53.620110 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.620030 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8"] Jul 30 14:55:53.621688 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:53.621666 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9868e2ce_3164_4d3e_9252_12f4bdeff3fc.slice/crio-95b5e17301182ac18d106bfbdfe5410bb9438151036ce5f3e7b0fafff3d086f7 WatchSource:0}: Error finding container 95b5e17301182ac18d106bfbdfe5410bb9438151036ce5f3e7b0fafff3d086f7: Status 404 returned error can't find the container with id 95b5e17301182ac18d106bfbdfe5410bb9438151036ce5f3e7b0fafff3d086f7 Jul 30 14:55:53.623135 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.623036 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg"] Jul 30 14:55:53.623305 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:55:53.623228 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd72a343e_ad27_471f_bf13_946695e4396e.slice/crio-ac2e0e6e645fc0644d9ccdfe6c6bf217c9b9c4cad2b90bb4e39b0961bd957abc WatchSource:0}: Error finding container ac2e0e6e645fc0644d9ccdfe6c6bf217c9b9c4cad2b90bb4e39b0961bd957abc: Status 404 returned error can't find the container with id ac2e0e6e645fc0644d9ccdfe6c6bf217c9b9c4cad2b90bb4e39b0961bd957abc Jul 30 14:55:53.939567 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.939488 2564 generic.go:356] "Generic (PLEG): container finished" podID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" containerID="55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b" exitCode=0 Jul 30 14:55:53.939567 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.939553 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:53.939889 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.939574 2564 scope.go:117] "RemoveContainer" containerID="55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b" Jul 30 14:55:53.941455 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.941429 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" event={"ID":"e6f8872c-8d81-4c86-9ae5-e33d0935a23d","Type":"ContainerStarted","Data":"74f77e3a98bb32a6e5a869e0ef6840cd2511db1b4283bcc8d405011651154a6f"} Jul 30 14:55:53.941717 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.941689 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:53.943241 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.943213 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:53.943372 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.943236 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" event={"ID":"3a86e7cb-c0ab-4858-9507-577203b2cbeb","Type":"ContainerStarted","Data":"80149baa31d75d304c042a5b59ddca7833a1715d2214066e1ad7cfa0d347e74e"} Jul 30 14:55:53.943372 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.943286 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" event={"ID":"3a86e7cb-c0ab-4858-9507-577203b2cbeb","Type":"ContainerStarted","Data":"3cd8defe9d6b1e48556e830687ba752985cf41357f6b22cfeee98a4e5da101f5"} Jul 30 14:55:53.943372 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.943319 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" Jul 30 14:55:53.945128 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.945105 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" event={"ID":"9868e2ce-3164-4d3e-9252-12f4bdeff3fc","Type":"ContainerStarted","Data":"3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6"} Jul 30 14:55:53.945234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.945148 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" event={"ID":"9868e2ce-3164-4d3e-9252-12f4bdeff3fc","Type":"ContainerStarted","Data":"95b5e17301182ac18d106bfbdfe5410bb9438151036ce5f3e7b0fafff3d086f7"} Jul 30 14:55:53.945234 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.945175 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:55:53.947097 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.946978 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" event={"ID":"d72a343e-ad27-471f-bf13-946695e4396e","Type":"ContainerStarted","Data":"012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079"} Jul 30 14:55:53.947097 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.947024 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" event={"ID":"d72a343e-ad27-471f-bf13-946695e4396e","Type":"ContainerStarted","Data":"ac2e0e6e645fc0644d9ccdfe6c6bf217c9b9c4cad2b90bb4e39b0961bd957abc"} Jul 30 14:55:53.947097 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.947052 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:55:53.948435 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.948413 2564 generic.go:356] "Generic (PLEG): container finished" podID="049e518e-d6df-428b-9e39-c92b5bebbab6" containerID="c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18" exitCode=0 Jul 30 14:55:53.948552 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.948464 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6db879668d-bhdwm" Jul 30 14:55:53.948552 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.948495 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db879668d-bhdwm" event={"ID":"049e518e-d6df-428b-9e39-c92b5bebbab6","Type":"ContainerDied","Data":"c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18"} Jul 30 14:55:53.948552 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.948534 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6db879668d-bhdwm" event={"ID":"049e518e-d6df-428b-9e39-c92b5bebbab6","Type":"ContainerDied","Data":"073c8a5bc599bb3bfed2efed31403b0c42634d4097b24029cd3866161c2ad1ea"} Jul 30 14:55:53.950370 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.950346 2564 scope.go:117] "RemoveContainer" containerID="55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b" Jul 30 14:55:53.950370 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.950353 2564 generic.go:356] "Generic (PLEG): container finished" podID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" containerID="9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e" exitCode=0 Jul 30 14:55:53.950470 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.950398 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:53.950643 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.950625 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b\": container with ID starting with 55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b not found: ID does not exist" containerID="55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b" Jul 30 14:55:53.950719 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.950650 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b"} err="failed to get container status \"55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b\": rpc error: code = NotFound desc = could not find container \"55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b\": container with ID starting with 55904b7f5bfbed76ccfd6166b0cdc894863b236c4dc1037e50ddc0afa9d4588b not found: ID does not exist" Jul 30 14:55:53.950719 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.950666 2564 scope.go:117] "RemoveContainer" containerID="c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18" Jul 30 14:55:53.960385 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.960333 2564 scope.go:117] "RemoveContainer" containerID="c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18" Jul 30 14:55:53.960719 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.960638 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18\": container with ID starting with c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18 not found: ID does not exist" containerID="c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18" Jul 30 14:55:53.960719 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.960672 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18"} err="failed to get container status \"c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18\": rpc error: code = NotFound desc = could not find container \"c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18\": container with ID starting with c1a400b3ad343e8ebbeb568ef69ba785cd655356776e72c4f405f9cae909be18 not found: ID does not exist" Jul 30 14:55:53.960719 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.960695 2564 scope.go:117] "RemoveContainer" containerID="9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e" Jul 30 14:55:53.962636 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.962600 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lw6gk" podStartSLOduration=1.6655608210000001 podStartE2EDuration="25.962587442s" totalImagesPullingTime="24.297026621s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:53.960816808 +0000 UTC m=+480.944100145" watchObservedRunningTime="2026-07-30 14:55:53.962587442 +0000 UTC m=+480.945870813" Jul 30 14:55:53.972382 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.972349 2564 scope.go:117] "RemoveContainer" containerID="9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e" Jul 30 14:55:53.972687 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:53.972661 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e\": container with ID starting with 9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e not found: ID does not exist" containerID="9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e" Jul 30 14:55:53.972847 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.972817 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e"} err="failed to get container status \"9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e\": rpc error: code = NotFound desc = could not find container \"9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e\": container with ID starting with 9a23cb506bdbf2f17343c8e0e54eed9f75584f7a961fba1cb904e83b00ada74e not found: ID does not exist" Jul 30 14:55:53.983295 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:53.983190 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" podStartSLOduration=15.983169878 podStartE2EDuration="15.983169878s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:53.982149901 +0000 UTC m=+480.965433237" watchObservedRunningTime="2026-07-30 14:55:53.983169878 +0000 UTC m=+480.966453217" Jul 30 14:55:54.003542 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:54.003508 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-nvpw9\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-nvpw9" Jul 30 14:55:54.003709 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:54.003624 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" podStartSLOduration=16.0036066 podStartE2EDuration="16.0036066s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:54.001646011 +0000 UTC m=+480.984929369" watchObservedRunningTime="2026-07-30 14:55:54.0036066 +0000 UTC m=+480.986889937" Jul 30 14:55:54.026835 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:55:54.026803 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-k2hsh\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-k2hsh" Jul 30 14:55:54.027990 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:54.027948 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" podStartSLOduration=16.027933197 podStartE2EDuration="16.027933197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:55:54.025142774 +0000 UTC m=+481.008426110" watchObservedRunningTime="2026-07-30 14:55:54.027933197 +0000 UTC m=+481.011216533" Jul 30 14:55:54.042842 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:54.042812 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6db879668d-bhdwm"] Jul 30 14:55:54.047492 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:54.047458 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6db879668d-bhdwm"] Jul 30 14:55:55.491889 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:55.491854 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="049e518e-d6df-428b-9e39-c92b5bebbab6" path="/var/lib/kubelet/pods/049e518e-d6df-428b-9e39-c92b5bebbab6/volumes" Jul 30 14:55:55.492420 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:55.492399 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fc0c698-ec3c-451b-a669-e8e9d50c29cc" path="/var/lib/kubelet/pods/6fc0c698-ec3c-451b-a669-e8e9d50c29cc/volumes" Jul 30 14:55:55.492761 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:55:55.492743 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c" path="/var/lib/kubelet/pods/8aa7920b-6a7f-47a0-b8d1-4116d4b44f9c/volumes" Jul 30 14:56:04.961505 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:04.961468 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-4vg2s" Jul 30 14:56:04.962040 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:04.961653 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:56:04.962040 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:04.961707 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 14:56:05.056101 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:05.056059 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8"] Jul 30 14:56:05.056334 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:05.056277 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" podUID="d72a343e-ad27-471f-bf13-946695e4396e" containerName="manager" containerID="cri-o://012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079" gracePeriod=10 Jul 30 14:56:05.805833 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:05.805808 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:56:05.933485 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:05.933391 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j96kf\" (UniqueName: \"kubernetes.io/projected/d72a343e-ad27-471f-bf13-946695e4396e-kube-api-access-j96kf\") pod \"d72a343e-ad27-471f-bf13-946695e4396e\" (UID: \"d72a343e-ad27-471f-bf13-946695e4396e\") " Jul 30 14:56:05.933485 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:05.933431 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d72a343e-ad27-471f-bf13-946695e4396e-extensions-socket-volume\") pod \"d72a343e-ad27-471f-bf13-946695e4396e\" (UID: \"d72a343e-ad27-471f-bf13-946695e4396e\") " Jul 30 14:56:05.933861 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:05.933837 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d72a343e-ad27-471f-bf13-946695e4396e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d72a343e-ad27-471f-bf13-946695e4396e" (UID: "d72a343e-ad27-471f-bf13-946695e4396e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 14:56:05.935589 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:05.935555 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d72a343e-ad27-471f-bf13-946695e4396e-kube-api-access-j96kf" (OuterVolumeSpecName: "kube-api-access-j96kf") pod "d72a343e-ad27-471f-bf13-946695e4396e" (UID: "d72a343e-ad27-471f-bf13-946695e4396e"). InnerVolumeSpecName "kube-api-access-j96kf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:56:06.030331 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.030293 2564 generic.go:356] "Generic (PLEG): container finished" podID="d72a343e-ad27-471f-bf13-946695e4396e" containerID="012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079" exitCode=0 Jul 30 14:56:06.030824 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.030337 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" Jul 30 14:56:06.030824 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.030383 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" event={"ID":"d72a343e-ad27-471f-bf13-946695e4396e","Type":"ContainerDied","Data":"012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079"} Jul 30 14:56:06.030824 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.030448 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8" event={"ID":"d72a343e-ad27-471f-bf13-946695e4396e","Type":"ContainerDied","Data":"ac2e0e6e645fc0644d9ccdfe6c6bf217c9b9c4cad2b90bb4e39b0961bd957abc"} Jul 30 14:56:06.030824 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.030492 2564 scope.go:117] "RemoveContainer" containerID="012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079" Jul 30 14:56:06.034309 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.034287 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j96kf\" (UniqueName: \"kubernetes.io/projected/d72a343e-ad27-471f-bf13-946695e4396e-kube-api-access-j96kf\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:56:06.034397 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.034313 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d72a343e-ad27-471f-bf13-946695e4396e-extensions-socket-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:56:06.043815 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.042203 2564 scope.go:117] "RemoveContainer" containerID="012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079" Jul 30 14:56:06.043815 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:56:06.042598 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079\": container with ID starting with 012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079 not found: ID does not exist" containerID="012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079" Jul 30 14:56:06.043815 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.042639 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079"} err="failed to get container status \"012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079\": rpc error: code = NotFound desc = could not find container \"012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079\": container with ID starting with 012f70f2c580802b28fe0a1e6aa4192ce59e6a7dd31e8b39b5a18cd3dd751079 not found: ID does not exist" Jul 30 14:56:06.057847 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.057817 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8"] Jul 30 14:56:06.065686 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:06.065659 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-nckh8"] Jul 30 14:56:07.491621 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:07.491587 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d72a343e-ad27-471f-bf13-946695e4396e" path="/var/lib/kubelet/pods/d72a343e-ad27-471f-bf13-946695e4396e/volumes" Jul 30 14:56:21.514273 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.514232 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv"] Jul 30 14:56:21.514921 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.514901 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d72a343e-ad27-471f-bf13-946695e4396e" containerName="manager" Jul 30 14:56:21.514972 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.514926 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d72a343e-ad27-471f-bf13-946695e4396e" containerName="manager" Jul 30 14:56:21.514972 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.514952 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="049e518e-d6df-428b-9e39-c92b5bebbab6" containerName="console" Jul 30 14:56:21.514972 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.514961 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="049e518e-d6df-428b-9e39-c92b5bebbab6" containerName="console" Jul 30 14:56:21.515100 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.515060 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="049e518e-d6df-428b-9e39-c92b5bebbab6" containerName="console" Jul 30 14:56:21.515100 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.515094 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d72a343e-ad27-471f-bf13-946695e4396e" containerName="manager" Jul 30 14:56:21.597183 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.597151 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv"] Jul 30 14:56:21.597350 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.597257 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.599372 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.599344 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-kgx2p\"" Jul 30 14:56:21.675747 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.675712 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.675958 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.675778 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42v49\" (UniqueName: \"kubernetes.io/projected/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-kube-api-access-42v49\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.675958 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.675889 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.676116 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.675968 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.676116 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.676001 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.676116 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.676042 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.676116 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.676096 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.676305 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.676184 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.676305 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.676219 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777507 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777423 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777507 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777496 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777550 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42v49\" (UniqueName: \"kubernetes.io/projected/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-kube-api-access-42v49\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777649 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777680 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777981 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777835 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777981 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777897 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777981 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777935 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.777981 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.778244 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.777984 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.778244 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.778166 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.778244 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.778215 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.778684 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.778660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.780416 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.780347 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.780642 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.780622 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.786208 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.786158 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.786317 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.786227 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-42v49\" (UniqueName: \"kubernetes.io/projected/7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61-kube-api-access-42v49\") pod \"maas-default-gateway-openshift-default-78bb78ff78-9sfjv\" (UID: \"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:21.908416 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:21.908379 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:22.102417 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:22.102392 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv"] Jul 30 14:56:22.104520 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:56:22.104492 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2f1253_4668_4d7e_9bf7_c8c2ac88cb61.slice/crio-875b57c4668eb4e850eedf3d621002888848fe0623c375dc82df2d2f2e0349e2 WatchSource:0}: Error finding container 875b57c4668eb4e850eedf3d621002888848fe0623c375dc82df2d2f2e0349e2: Status 404 returned error can't find the container with id 875b57c4668eb4e850eedf3d621002888848fe0623c375dc82df2d2f2e0349e2 Jul 30 14:56:22.106718 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:22.106687 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:56:22.106791 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:22.106751 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:56:22.106791 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:22.106785 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:56:22.132399 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:22.132369 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" event={"ID":"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61","Type":"ContainerStarted","Data":"875b57c4668eb4e850eedf3d621002888848fe0623c375dc82df2d2f2e0349e2"} Jul 30 14:56:23.146778 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:23.146734 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" event={"ID":"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61","Type":"ContainerStarted","Data":"a0bf4ba7b9fd98358229787821ec5b77a7c147d10873cb0012353389ec38055d"} Jul 30 14:56:23.174792 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:23.174726 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" podStartSLOduration=2.174705449 podStartE2EDuration="2.174705449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:56:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:56:23.17396078 +0000 UTC m=+510.157244116" watchObservedRunningTime="2026-07-30 14:56:23.174705449 +0000 UTC m=+510.157988786" Jul 30 14:56:23.908835 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:23.908799 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:23.913158 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:23.913131 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:24.153132 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:24.153104 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:24.153897 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:24.153878 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:56:25.124015 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.123981 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-2f6xz"] Jul 30 14:56:25.155454 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.155423 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-2f6xz"] Jul 30 14:56:25.155907 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.155530 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2f6xz" Jul 30 14:56:25.157714 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.157688 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-m5wft\"" Jul 30 14:56:25.207231 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.207195 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7qlv\" (UniqueName: \"kubernetes.io/projected/bc2b186b-c693-4d37-b825-a26401398756-kube-api-access-m7qlv\") pod \"authorino-58cd955675-2f6xz\" (UID: \"bc2b186b-c693-4d37-b825-a26401398756\") " pod="kuadrant-system/authorino-58cd955675-2f6xz" Jul 30 14:56:25.307901 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.307871 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7qlv\" (UniqueName: \"kubernetes.io/projected/bc2b186b-c693-4d37-b825-a26401398756-kube-api-access-m7qlv\") pod \"authorino-58cd955675-2f6xz\" (UID: \"bc2b186b-c693-4d37-b825-a26401398756\") " pod="kuadrant-system/authorino-58cd955675-2f6xz" Jul 30 14:56:25.323294 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.323263 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-gv28k"] Jul 30 14:56:25.324160 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.324130 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7qlv\" (UniqueName: \"kubernetes.io/projected/bc2b186b-c693-4d37-b825-a26401398756-kube-api-access-m7qlv\") pod \"authorino-58cd955675-2f6xz\" (UID: \"bc2b186b-c693-4d37-b825-a26401398756\") " pod="kuadrant-system/authorino-58cd955675-2f6xz" Jul 30 14:56:25.327261 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.327241 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gv28k" Jul 30 14:56:25.331841 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.331818 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gv28k"] Jul 30 14:56:25.408564 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.408476 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crtd7\" (UniqueName: \"kubernetes.io/projected/5a752cce-12cd-4a7e-a30f-882aad6e3f46-kube-api-access-crtd7\") pod \"authorino-75d556bd77-gv28k\" (UID: \"5a752cce-12cd-4a7e-a30f-882aad6e3f46\") " pod="kuadrant-system/authorino-75d556bd77-gv28k" Jul 30 14:56:25.468476 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.468442 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2f6xz" Jul 30 14:56:25.509849 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.509816 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crtd7\" (UniqueName: \"kubernetes.io/projected/5a752cce-12cd-4a7e-a30f-882aad6e3f46-kube-api-access-crtd7\") pod \"authorino-75d556bd77-gv28k\" (UID: \"5a752cce-12cd-4a7e-a30f-882aad6e3f46\") " pod="kuadrant-system/authorino-75d556bd77-gv28k" Jul 30 14:56:25.519959 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.519903 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crtd7\" (UniqueName: \"kubernetes.io/projected/5a752cce-12cd-4a7e-a30f-882aad6e3f46-kube-api-access-crtd7\") pod \"authorino-75d556bd77-gv28k\" (UID: \"5a752cce-12cd-4a7e-a30f-882aad6e3f46\") " pod="kuadrant-system/authorino-75d556bd77-gv28k" Jul 30 14:56:25.615485 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.615407 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-2f6xz"] Jul 30 14:56:25.618172 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:56:25.618140 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc2b186b_c693_4d37_b825_a26401398756.slice/crio-37b4a58f88776d79856c2d7292ad3a13eca119840a38b3a3f77308ec317677df WatchSource:0}: Error finding container 37b4a58f88776d79856c2d7292ad3a13eca119840a38b3a3f77308ec317677df: Status 404 returned error can't find the container with id 37b4a58f88776d79856c2d7292ad3a13eca119840a38b3a3f77308ec317677df Jul 30 14:56:25.642712 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.642687 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gv28k" Jul 30 14:56:25.776730 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:25.776704 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gv28k"] Jul 30 14:56:25.778980 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:56:25.778957 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5a752cce_12cd_4a7e_a30f_882aad6e3f46.slice/crio-125521cf281b96fc1caff7576d347cbf706b2a135640f7c51a80aef2e3c66767 WatchSource:0}: Error finding container 125521cf281b96fc1caff7576d347cbf706b2a135640f7c51a80aef2e3c66767: Status 404 returned error can't find the container with id 125521cf281b96fc1caff7576d347cbf706b2a135640f7c51a80aef2e3c66767 Jul 30 14:56:26.166519 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:26.166467 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2f6xz" event={"ID":"bc2b186b-c693-4d37-b825-a26401398756","Type":"ContainerStarted","Data":"37b4a58f88776d79856c2d7292ad3a13eca119840a38b3a3f77308ec317677df"} Jul 30 14:56:26.167741 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:26.167718 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gv28k" event={"ID":"5a752cce-12cd-4a7e-a30f-882aad6e3f46","Type":"ContainerStarted","Data":"125521cf281b96fc1caff7576d347cbf706b2a135640f7c51a80aef2e3c66767"} Jul 30 14:56:30.201568 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:30.201527 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gv28k" event={"ID":"5a752cce-12cd-4a7e-a30f-882aad6e3f46","Type":"ContainerStarted","Data":"430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381"} Jul 30 14:56:30.203118 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:30.203087 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2f6xz" event={"ID":"bc2b186b-c693-4d37-b825-a26401398756","Type":"ContainerStarted","Data":"13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881"} Jul 30 14:56:30.218869 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:30.218817 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-gv28k" podStartSLOduration=1.6871987160000002 podStartE2EDuration="5.218801979s" totalImagesPullingTime="3.531603263s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:56:30.217846837 +0000 UTC m=+517.201130173" watchObservedRunningTime="2026-07-30 14:56:30.218801979 +0000 UTC m=+517.202085316" Jul 30 14:56:30.234213 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:30.234156 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-2f6xz" podStartSLOduration=1.558167099 podStartE2EDuration="5.234138352s" totalImagesPullingTime="3.675971253s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:56:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:56:30.231046523 +0000 UTC m=+517.214329862" watchObservedRunningTime="2026-07-30 14:56:30.234138352 +0000 UTC m=+517.217421688" Jul 30 14:56:30.259789 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:30.259752 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-2f6xz"] Jul 30 14:56:32.215057 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:32.215018 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-2f6xz" podUID="bc2b186b-c693-4d37-b825-a26401398756" containerName="authorino" containerID="cri-o://13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881" gracePeriod=30 Jul 30 14:56:32.466614 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:32.466550 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2f6xz" Jul 30 14:56:32.585903 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:32.585865 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m7qlv\" (UniqueName: \"kubernetes.io/projected/bc2b186b-c693-4d37-b825-a26401398756-kube-api-access-m7qlv\") pod \"bc2b186b-c693-4d37-b825-a26401398756\" (UID: \"bc2b186b-c693-4d37-b825-a26401398756\") " Jul 30 14:56:32.588003 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:32.587976 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc2b186b-c693-4d37-b825-a26401398756-kube-api-access-m7qlv" (OuterVolumeSpecName: "kube-api-access-m7qlv") pod "bc2b186b-c693-4d37-b825-a26401398756" (UID: "bc2b186b-c693-4d37-b825-a26401398756"). InnerVolumeSpecName "kube-api-access-m7qlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:56:32.687320 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:32.687283 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7qlv\" (UniqueName: \"kubernetes.io/projected/bc2b186b-c693-4d37-b825-a26401398756-kube-api-access-m7qlv\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:56:33.222786 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.222750 2564 generic.go:356] "Generic (PLEG): container finished" podID="bc2b186b-c693-4d37-b825-a26401398756" containerID="13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881" exitCode=0 Jul 30 14:56:33.222786 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.222788 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-2f6xz" Jul 30 14:56:33.223253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.222840 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2f6xz" event={"ID":"bc2b186b-c693-4d37-b825-a26401398756","Type":"ContainerDied","Data":"13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881"} Jul 30 14:56:33.223253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.222886 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-2f6xz" event={"ID":"bc2b186b-c693-4d37-b825-a26401398756","Type":"ContainerDied","Data":"37b4a58f88776d79856c2d7292ad3a13eca119840a38b3a3f77308ec317677df"} Jul 30 14:56:33.223253 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.222906 2564 scope.go:117] "RemoveContainer" containerID="13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881" Jul 30 14:56:33.231989 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.231970 2564 scope.go:117] "RemoveContainer" containerID="13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881" Jul 30 14:56:33.232283 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:56:33.232264 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881\": container with ID starting with 13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881 not found: ID does not exist" containerID="13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881" Jul 30 14:56:33.232339 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.232293 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881"} err="failed to get container status \"13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881\": rpc error: code = NotFound desc = could not find container \"13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881\": container with ID starting with 13579dc0386c46d53f33e208c6e2d669b6ed8b3af0047522305677c7a5b19881 not found: ID does not exist" Jul 30 14:56:33.243951 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.243926 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-2f6xz"] Jul 30 14:56:33.247384 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.247361 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-2f6xz"] Jul 30 14:56:33.490365 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:33.490330 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc2b186b-c693-4d37-b825-a26401398756" path="/var/lib/kubelet/pods/bc2b186b-c693-4d37-b825-a26401398756/volumes" Jul 30 14:56:58.779915 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.779873 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-262ds"] Jul 30 14:56:58.780510 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.780490 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc2b186b-c693-4d37-b825-a26401398756" containerName="authorino" Jul 30 14:56:58.780578 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.780511 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc2b186b-c693-4d37-b825-a26401398756" containerName="authorino" Jul 30 14:56:58.780615 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.780577 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc2b186b-c693-4d37-b825-a26401398756" containerName="authorino" Jul 30 14:56:58.785211 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.785192 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-262ds" Jul 30 14:56:58.790681 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.790652 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-262ds"] Jul 30 14:56:58.934501 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.934453 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89k79\" (UniqueName: \"kubernetes.io/projected/20a80349-40ce-4cbd-9457-35be09054120-kube-api-access-89k79\") pod \"authorino-5d4b67c446-262ds\" (UID: \"20a80349-40ce-4cbd-9457-35be09054120\") " pod="kuadrant-system/authorino-5d4b67c446-262ds" Jul 30 14:56:58.970649 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.970610 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-262ds"] Jul 30 14:56:58.970935 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:56:58.970915 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-89k79], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-262ds" podUID="20a80349-40ce-4cbd-9457-35be09054120" Jul 30 14:56:59.000029 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:58.999995 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-7dcs2"] Jul 30 14:56:59.003541 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.003522 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.005646 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.005626 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 30 14:56:59.013393 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.013367 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-7dcs2"] Jul 30 14:56:59.035547 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.035459 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k79\" (UniqueName: \"kubernetes.io/projected/20a80349-40ce-4cbd-9457-35be09054120-kube-api-access-89k79\") pod \"authorino-5d4b67c446-262ds\" (UID: \"20a80349-40ce-4cbd-9457-35be09054120\") " pod="kuadrant-system/authorino-5d4b67c446-262ds" Jul 30 14:56:59.035547 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.035512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b551296d-1043-4311-845a-21b3c5b52f44-tls-cert\") pod \"authorino-76877994-7dcs2\" (UID: \"b551296d-1043-4311-845a-21b3c5b52f44\") " pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.035547 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.035530 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbm4f\" (UniqueName: \"kubernetes.io/projected/b551296d-1043-4311-845a-21b3c5b52f44-kube-api-access-hbm4f\") pod \"authorino-76877994-7dcs2\" (UID: \"b551296d-1043-4311-845a-21b3c5b52f44\") " pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.044328 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.044298 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-7dcs2"] Jul 30 14:56:59.044601 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:56:59.044582 2564 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-hbm4f tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-76877994-7dcs2" podUID="b551296d-1043-4311-845a-21b3c5b52f44" Jul 30 14:56:59.046447 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.046424 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-89k79\" (UniqueName: \"kubernetes.io/projected/20a80349-40ce-4cbd-9457-35be09054120-kube-api-access-89k79\") pod \"authorino-5d4b67c446-262ds\" (UID: \"20a80349-40ce-4cbd-9457-35be09054120\") " pod="kuadrant-system/authorino-5d4b67c446-262ds" Jul 30 14:56:59.070641 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.070603 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5477478cb-k598d"] Jul 30 14:56:59.074312 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.074285 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 14:56:59.080792 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.080765 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5477478cb-k598d"] Jul 30 14:56:59.136204 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.136169 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdxwz\" (UniqueName: \"kubernetes.io/projected/e2fcbddc-a73d-4755-ad55-4725b26d7829-kube-api-access-wdxwz\") pod \"authorino-5477478cb-k598d\" (UID: \"e2fcbddc-a73d-4755-ad55-4725b26d7829\") " pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 14:56:59.136403 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.136214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b551296d-1043-4311-845a-21b3c5b52f44-tls-cert\") pod \"authorino-76877994-7dcs2\" (UID: \"b551296d-1043-4311-845a-21b3c5b52f44\") " pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.136403 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.136236 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbm4f\" (UniqueName: \"kubernetes.io/projected/b551296d-1043-4311-845a-21b3c5b52f44-kube-api-access-hbm4f\") pod \"authorino-76877994-7dcs2\" (UID: \"b551296d-1043-4311-845a-21b3c5b52f44\") " pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.136403 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.136315 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2fcbddc-a73d-4755-ad55-4725b26d7829-tls-cert\") pod \"authorino-5477478cb-k598d\" (UID: \"e2fcbddc-a73d-4755-ad55-4725b26d7829\") " pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 14:56:59.138654 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.138624 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b551296d-1043-4311-845a-21b3c5b52f44-tls-cert\") pod \"authorino-76877994-7dcs2\" (UID: \"b551296d-1043-4311-845a-21b3c5b52f44\") " pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.144115 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.144056 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbm4f\" (UniqueName: \"kubernetes.io/projected/b551296d-1043-4311-845a-21b3c5b52f44-kube-api-access-hbm4f\") pod \"authorino-76877994-7dcs2\" (UID: \"b551296d-1043-4311-845a-21b3c5b52f44\") " pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.237059 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.237023 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2fcbddc-a73d-4755-ad55-4725b26d7829-tls-cert\") pod \"authorino-5477478cb-k598d\" (UID: \"e2fcbddc-a73d-4755-ad55-4725b26d7829\") " pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 14:56:59.237235 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.237068 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdxwz\" (UniqueName: \"kubernetes.io/projected/e2fcbddc-a73d-4755-ad55-4725b26d7829-kube-api-access-wdxwz\") pod \"authorino-5477478cb-k598d\" (UID: \"e2fcbddc-a73d-4755-ad55-4725b26d7829\") " pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 14:56:59.239520 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.239490 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2fcbddc-a73d-4755-ad55-4725b26d7829-tls-cert\") pod \"authorino-5477478cb-k598d\" (UID: \"e2fcbddc-a73d-4755-ad55-4725b26d7829\") " pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 14:56:59.244372 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.244345 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdxwz\" (UniqueName: \"kubernetes.io/projected/e2fcbddc-a73d-4755-ad55-4725b26d7829-kube-api-access-wdxwz\") pod \"authorino-5477478cb-k598d\" (UID: \"e2fcbddc-a73d-4755-ad55-4725b26d7829\") " pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 14:56:59.374844 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.374747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.374844 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.374747 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-262ds" Jul 30 14:56:59.382738 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.382714 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:56:59.385185 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.385163 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 14:56:59.385951 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.385926 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-262ds" Jul 30 14:56:59.438172 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.438140 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-89k79\" (UniqueName: \"kubernetes.io/projected/20a80349-40ce-4cbd-9457-35be09054120-kube-api-access-89k79\") pod \"20a80349-40ce-4cbd-9457-35be09054120\" (UID: \"20a80349-40ce-4cbd-9457-35be09054120\") " Jul 30 14:56:59.438172 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.438173 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hbm4f\" (UniqueName: \"kubernetes.io/projected/b551296d-1043-4311-845a-21b3c5b52f44-kube-api-access-hbm4f\") pod \"b551296d-1043-4311-845a-21b3c5b52f44\" (UID: \"b551296d-1043-4311-845a-21b3c5b52f44\") " Jul 30 14:56:59.438403 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.438239 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b551296d-1043-4311-845a-21b3c5b52f44-tls-cert\") pod \"b551296d-1043-4311-845a-21b3c5b52f44\" (UID: \"b551296d-1043-4311-845a-21b3c5b52f44\") " Jul 30 14:56:59.440759 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.440697 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b551296d-1043-4311-845a-21b3c5b52f44-kube-api-access-hbm4f" (OuterVolumeSpecName: "kube-api-access-hbm4f") pod "b551296d-1043-4311-845a-21b3c5b52f44" (UID: "b551296d-1043-4311-845a-21b3c5b52f44"). InnerVolumeSpecName "kube-api-access-hbm4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:56:59.440759 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.440708 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a80349-40ce-4cbd-9457-35be09054120-kube-api-access-89k79" (OuterVolumeSpecName: "kube-api-access-89k79") pod "20a80349-40ce-4cbd-9457-35be09054120" (UID: "20a80349-40ce-4cbd-9457-35be09054120"). InnerVolumeSpecName "kube-api-access-89k79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:56:59.440759 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.440706 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b551296d-1043-4311-845a-21b3c5b52f44-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "b551296d-1043-4311-845a-21b3c5b52f44" (UID: "b551296d-1043-4311-845a-21b3c5b52f44"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 14:56:59.541241 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.539974 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b551296d-1043-4311-845a-21b3c5b52f44-tls-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:56:59.541241 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.540018 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89k79\" (UniqueName: \"kubernetes.io/projected/20a80349-40ce-4cbd-9457-35be09054120-kube-api-access-89k79\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:56:59.541241 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.540036 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hbm4f\" (UniqueName: \"kubernetes.io/projected/b551296d-1043-4311-845a-21b3c5b52f44-kube-api-access-hbm4f\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:56:59.547592 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:56:59.546700 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5477478cb-k598d"] Jul 30 14:56:59.553377 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:56:59.553351 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode2fcbddc_a73d_4755_ad55_4725b26d7829.slice/crio-1a7d417f778f8a47314b91ad9f70e8e4d2845fb7fa15ebd19a1db51cd9f606f7 WatchSource:0}: Error finding container 1a7d417f778f8a47314b91ad9f70e8e4d2845fb7fa15ebd19a1db51cd9f606f7: Status 404 returned error can't find the container with id 1a7d417f778f8a47314b91ad9f70e8e4d2845fb7fa15ebd19a1db51cd9f606f7 Jul 30 14:57:00.381583 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.381538 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5477478cb-k598d" event={"ID":"e2fcbddc-a73d-4755-ad55-4725b26d7829","Type":"ContainerStarted","Data":"451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11"} Jul 30 14:57:00.381583 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.381577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-7dcs2" Jul 30 14:57:00.382045 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.381594 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5477478cb-k598d" event={"ID":"e2fcbddc-a73d-4755-ad55-4725b26d7829","Type":"ContainerStarted","Data":"1a7d417f778f8a47314b91ad9f70e8e4d2845fb7fa15ebd19a1db51cd9f606f7"} Jul 30 14:57:00.382045 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.381577 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-262ds" Jul 30 14:57:00.400346 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.400297 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5477478cb-k598d" podStartSLOduration=1.052955998 podStartE2EDuration="1.400281258s" totalImagesPullingTime="347.32526ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:56:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:57:00.397923151 +0000 UTC m=+547.381206489" watchObservedRunningTime="2026-07-30 14:57:00.400281258 +0000 UTC m=+547.383564595" Jul 30 14:57:00.428278 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.428237 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gv28k"] Jul 30 14:57:00.428801 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.428744 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-gv28k" podUID="5a752cce-12cd-4a7e-a30f-882aad6e3f46" containerName="authorino" containerID="cri-o://430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381" gracePeriod=30 Jul 30 14:57:00.432445 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.432407 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-262ds"] Jul 30 14:57:00.439680 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.439649 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-262ds"] Jul 30 14:57:00.461300 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.461218 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-7dcs2"] Jul 30 14:57:00.465351 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.465322 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-7dcs2"] Jul 30 14:57:00.696165 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.696136 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gv28k" Jul 30 14:57:00.750504 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.750465 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crtd7\" (UniqueName: \"kubernetes.io/projected/5a752cce-12cd-4a7e-a30f-882aad6e3f46-kube-api-access-crtd7\") pod \"5a752cce-12cd-4a7e-a30f-882aad6e3f46\" (UID: \"5a752cce-12cd-4a7e-a30f-882aad6e3f46\") " Jul 30 14:57:00.752555 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.752525 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a752cce-12cd-4a7e-a30f-882aad6e3f46-kube-api-access-crtd7" (OuterVolumeSpecName: "kube-api-access-crtd7") pod "5a752cce-12cd-4a7e-a30f-882aad6e3f46" (UID: "5a752cce-12cd-4a7e-a30f-882aad6e3f46"). InnerVolumeSpecName "kube-api-access-crtd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 14:57:00.851526 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:00.851487 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crtd7\" (UniqueName: \"kubernetes.io/projected/5a752cce-12cd-4a7e-a30f-882aad6e3f46-kube-api-access-crtd7\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 14:57:01.389135 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.389101 2564 generic.go:356] "Generic (PLEG): container finished" podID="5a752cce-12cd-4a7e-a30f-882aad6e3f46" containerID="430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381" exitCode=0 Jul 30 14:57:01.389708 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.389189 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gv28k" Jul 30 14:57:01.389708 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.389245 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gv28k" event={"ID":"5a752cce-12cd-4a7e-a30f-882aad6e3f46","Type":"ContainerDied","Data":"430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381"} Jul 30 14:57:01.389708 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.389299 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gv28k" event={"ID":"5a752cce-12cd-4a7e-a30f-882aad6e3f46","Type":"ContainerDied","Data":"125521cf281b96fc1caff7576d347cbf706b2a135640f7c51a80aef2e3c66767"} Jul 30 14:57:01.389708 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.389323 2564 scope.go:117] "RemoveContainer" containerID="430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381" Jul 30 14:57:01.399094 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.399051 2564 scope.go:117] "RemoveContainer" containerID="430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381" Jul 30 14:57:01.399434 ip-10-0-130-38 kubenswrapper[2564]: E0730 14:57:01.399415 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381\": container with ID starting with 430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381 not found: ID does not exist" containerID="430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381" Jul 30 14:57:01.399484 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.399444 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381"} err="failed to get container status \"430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381\": rpc error: code = NotFound desc = could not find container \"430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381\": container with ID starting with 430a87b85910aea09c41583d8ad234e962a91ccf1fb760805c8a5e9ee16d2381 not found: ID does not exist" Jul 30 14:57:01.414155 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.414119 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gv28k"] Jul 30 14:57:01.420168 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.420132 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gv28k"] Jul 30 14:57:01.491232 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.491196 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20a80349-40ce-4cbd-9457-35be09054120" path="/var/lib/kubelet/pods/20a80349-40ce-4cbd-9457-35be09054120/volumes" Jul 30 14:57:01.491469 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.491456 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a752cce-12cd-4a7e-a30f-882aad6e3f46" path="/var/lib/kubelet/pods/5a752cce-12cd-4a7e-a30f-882aad6e3f46/volumes" Jul 30 14:57:01.491797 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:01.491785 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b551296d-1043-4311-845a-21b3c5b52f44" path="/var/lib/kubelet/pods/b551296d-1043-4311-845a-21b3c5b52f44/volumes" Jul 30 14:57:43.129343 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.129305 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk"] Jul 30 14:57:43.129976 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.129956 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a752cce-12cd-4a7e-a30f-882aad6e3f46" containerName="authorino" Jul 30 14:57:43.130063 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.129979 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a752cce-12cd-4a7e-a30f-882aad6e3f46" containerName="authorino" Jul 30 14:57:43.130171 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.130089 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a752cce-12cd-4a7e-a30f-882aad6e3f46" containerName="authorino" Jul 30 14:57:43.132474 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.132455 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:43.134662 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.134638 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-lt24z\"" Jul 30 14:57:43.134939 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.134918 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 30 14:57:43.143687 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.143656 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jg92\" (UniqueName: \"kubernetes.io/projected/1ffefa41-b18b-4eee-9c48-6f0bba8f2f18-kube-api-access-8jg92\") pod \"payload-pre-processing-687dbb6cf6-wj2pk\" (UID: \"1ffefa41-b18b-4eee-9c48-6f0bba8f2f18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:43.143842 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.143819 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffefa41-b18b-4eee-9c48-6f0bba8f2f18-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wj2pk\" (UID: \"1ffefa41-b18b-4eee-9c48-6f0bba8f2f18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:43.146172 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.146150 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk"] Jul 30 14:57:43.244767 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.244730 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jg92\" (UniqueName: \"kubernetes.io/projected/1ffefa41-b18b-4eee-9c48-6f0bba8f2f18-kube-api-access-8jg92\") pod \"payload-pre-processing-687dbb6cf6-wj2pk\" (UID: \"1ffefa41-b18b-4eee-9c48-6f0bba8f2f18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:43.244938 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.244826 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffefa41-b18b-4eee-9c48-6f0bba8f2f18-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wj2pk\" (UID: \"1ffefa41-b18b-4eee-9c48-6f0bba8f2f18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:43.245550 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.245529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ffefa41-b18b-4eee-9c48-6f0bba8f2f18-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-wj2pk\" (UID: \"1ffefa41-b18b-4eee-9c48-6f0bba8f2f18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:43.256351 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.256314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jg92\" (UniqueName: \"kubernetes.io/projected/1ffefa41-b18b-4eee-9c48-6f0bba8f2f18-kube-api-access-8jg92\") pod \"payload-pre-processing-687dbb6cf6-wj2pk\" (UID: \"1ffefa41-b18b-4eee-9c48-6f0bba8f2f18\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:43.444606 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.444514 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:43.806944 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:43.806902 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk"] Jul 30 14:57:43.814699 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:57:43.814661 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffefa41_b18b_4eee_9c48_6f0bba8f2f18.slice/crio-a97262acaeb9a4120b49614a90c1a5d12780be577484414e30e2d01da9c85f0a WatchSource:0}: Error finding container a97262acaeb9a4120b49614a90c1a5d12780be577484414e30e2d01da9c85f0a: Status 404 returned error can't find the container with id a97262acaeb9a4120b49614a90c1a5d12780be577484414e30e2d01da9c85f0a Jul 30 14:57:44.641303 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:44.641265 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" event={"ID":"1ffefa41-b18b-4eee-9c48-6f0bba8f2f18","Type":"ContainerStarted","Data":"a97262acaeb9a4120b49614a90c1a5d12780be577484414e30e2d01da9c85f0a"} Jul 30 14:57:46.655764 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:46.655721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" event={"ID":"1ffefa41-b18b-4eee-9c48-6f0bba8f2f18","Type":"ContainerStarted","Data":"a78e70cf73c631eedb75d498a6dc093eda4c68eb2a074d9e12a469a2878a4e7a"} Jul 30 14:57:46.656205 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:46.655811 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:57:46.673917 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:46.673843 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" podStartSLOduration=1.751428551 podStartE2EDuration="3.673821556s" totalImagesPullingTime="1.922393005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:57:46.669833363 +0000 UTC m=+593.653116727" watchObservedRunningTime="2026-07-30 14:57:46.673821556 +0000 UTC m=+593.657104898" Jul 30 14:57:51.688788 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:51.688751 2564 generic.go:356] "Generic (PLEG): container finished" podID="7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61" containerID="a0bf4ba7b9fd98358229787821ec5b77a7c147d10873cb0012353389ec38055d" exitCode=0 Jul 30 14:57:51.689256 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:51.688808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" event={"ID":"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61","Type":"ContainerDied","Data":"a0bf4ba7b9fd98358229787821ec5b77a7c147d10873cb0012353389ec38055d"} Jul 30 14:57:51.689256 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:51.689212 2564 scope.go:117] "RemoveContainer" containerID="a0bf4ba7b9fd98358229787821ec5b77a7c147d10873cb0012353389ec38055d" Jul 30 14:57:51.690272 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:51.690235 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:57:51.690377 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:51.690325 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:57:51.690431 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:51.690371 2564 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"} Jul 30 14:57:52.697654 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:52.697615 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" event={"ID":"7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61","Type":"ContainerStarted","Data":"88c6263f48755df32ca6cf2cbf6aa7cb9cd96616c6a0f34de17ea844918d0be5"} Jul 30 14:57:52.908849 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:52.908817 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:57:52.909623 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:52.909594 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9sfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 30 14:57:52.909748 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:52.909643 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" podUID="7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 30 14:57:53.518047 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:53.518016 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 14:57:53.518273 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:53.518102 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 14:57:53.909267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:53.909176 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9sfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 30 14:57:53.909267 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:53.909248 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" podUID="7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 30 14:57:54.153167 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:54.153130 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:57:54.909484 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:54.909448 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9sfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 30 14:57:54.909910 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:54.909527 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" podUID="7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 30 14:57:55.909715 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:55.909679 2564 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-9sfjv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" start-of-body= Jul 30 14:57:55.910167 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:55.909737 2564 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" podUID="7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.54:15021/healthz/ready\": dial tcp 10.132.0.54:15021: connect: connection refused" Jul 30 14:57:56.913296 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:56.913267 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:57:56.914343 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:56.914323 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-9sfjv" Jul 30 14:57:57.665803 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:57:57.665770 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-wj2pk" Jul 30 14:58:11.945772 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:11.945735 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr"] Jul 30 14:58:11.950753 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:11.950727 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:11.953886 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:11.953863 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 30 14:58:11.954015 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:11.953872 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-756b8\"" Jul 30 14:58:11.954015 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:11.953920 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 30 14:58:11.954015 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:11.953872 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 30 14:58:11.959542 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:11.959520 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr"] Jul 30 14:58:12.046993 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.046955 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.047237 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.047010 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.047237 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.047155 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768617bf-fad6-475c-b760-5223bed08502-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.047237 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.047223 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.047401 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.047268 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gddm4\" (UniqueName: \"kubernetes.io/projected/768617bf-fad6-475c-b760-5223bed08502-kube-api-access-gddm4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.047401 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.047354 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148098 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148056 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148302 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148113 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gddm4\" (UniqueName: \"kubernetes.io/projected/768617bf-fad6-475c-b760-5223bed08502-kube-api-access-gddm4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148302 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148154 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148302 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148185 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148302 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148204 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148302 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148245 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768617bf-fad6-475c-b760-5223bed08502-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148588 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148549 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148634 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148598 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.148634 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.148611 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.150542 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.150516 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/768617bf-fad6-475c-b760-5223bed08502-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.150842 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.150819 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/768617bf-fad6-475c-b760-5223bed08502-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.156910 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.156884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gddm4\" (UniqueName: \"kubernetes.io/projected/768617bf-fad6-475c-b760-5223bed08502-kube-api-access-gddm4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr\" (UID: \"768617bf-fad6-475c-b760-5223bed08502\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.263570 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.263529 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:12.423065 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.423031 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr"] Jul 30 14:58:12.425531 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:58:12.425504 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod768617bf_fad6_475c_b760_5223bed08502.slice/crio-e4a893152bf47346beacfe6ac70b10bd8b229d399abf534453f5ecd07a8b7657 WatchSource:0}: Error finding container e4a893152bf47346beacfe6ac70b10bd8b229d399abf534453f5ecd07a8b7657: Status 404 returned error can't find the container with id e4a893152bf47346beacfe6ac70b10bd8b229d399abf534453f5ecd07a8b7657 Jul 30 14:58:12.427368 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.427346 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 14:58:12.818889 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:12.818808 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" event={"ID":"768617bf-fad6-475c-b760-5223bed08502","Type":"ContainerStarted","Data":"e4a893152bf47346beacfe6ac70b10bd8b229d399abf534453f5ecd07a8b7657"} Jul 30 14:58:13.833892 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:13.833853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" event={"ID":"768617bf-fad6-475c-b760-5223bed08502","Type":"ContainerStarted","Data":"ec37870a9d70f091b96775683302747e303724a35cc9908043135d4a5420a0b4"} Jul 30 14:58:13.853150 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:13.853035 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" podStartSLOduration=1.677270687 podStartE2EDuration="2.853017084s" totalImagesPullingTime="1.175746397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:58:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:58:13.85077245 +0000 UTC m=+620.834055784" watchObservedRunningTime="2026-07-30 14:58:13.853017084 +0000 UTC m=+620.836300465" Jul 30 14:58:22.264474 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:22.264385 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:22.264474 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:22.264453 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:22.277453 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:22.277421 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:22.897958 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:22.897928 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr" Jul 30 14:58:42.731159 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.731110 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz"] Jul 30 14:58:42.735047 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.735026 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.737911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.737893 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Jul 30 14:58:42.750055 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.750027 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz"] Jul 30 14:58:42.856247 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.856210 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.856426 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.856273 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.856426 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.856347 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s9zt\" (UniqueName: \"kubernetes.io/projected/2a720ed2-9b7f-4615-a3be-6c9437b47f54-kube-api-access-6s9zt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.856426 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.856380 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.856426 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.856405 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a720ed2-9b7f-4615-a3be-6c9437b47f54-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.856426 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.856422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.957572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.957543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a720ed2-9b7f-4615-a3be-6c9437b47f54-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.957572 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.957574 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.957835 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.957632 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.957835 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.957670 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.957835 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.957711 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s9zt\" (UniqueName: \"kubernetes.io/projected/2a720ed2-9b7f-4615-a3be-6c9437b47f54-kube-api-access-6s9zt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.957835 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.957748 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.958182 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.958145 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.958332 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.958196 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.958332 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.958263 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.960056 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.960034 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a720ed2-9b7f-4615-a3be-6c9437b47f54-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.960233 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.960214 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a720ed2-9b7f-4615-a3be-6c9437b47f54-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:42.968181 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:42.968151 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s9zt\" (UniqueName: \"kubernetes.io/projected/2a720ed2-9b7f-4615-a3be-6c9437b47f54-kube-api-access-6s9zt\") pod \"e2e-trlp-test-simulated-kserve-598466548d-5hzdz\" (UID: \"2a720ed2-9b7f-4615-a3be-6c9437b47f54\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:43.045003 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:43.044916 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:43.192935 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:43.192904 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz"] Jul 30 14:58:43.195897 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:58:43.195871 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a720ed2_9b7f_4615_a3be_6c9437b47f54.slice/crio-6993823c8f99c143f73369a656849011528c5b0b32bcf43b915607a67031f3a2 WatchSource:0}: Error finding container 6993823c8f99c143f73369a656849011528c5b0b32bcf43b915607a67031f3a2: Status 404 returned error can't find the container with id 6993823c8f99c143f73369a656849011528c5b0b32bcf43b915607a67031f3a2 Jul 30 14:58:44.009511 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:44.009475 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" event={"ID":"2a720ed2-9b7f-4615-a3be-6c9437b47f54","Type":"ContainerStarted","Data":"a019d70deff3c4dbb37d0c331fa8a95864615479c6fea82e0887ac088922c604"} Jul 30 14:58:44.009911 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:44.009535 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" event={"ID":"2a720ed2-9b7f-4615-a3be-6c9437b47f54","Type":"ContainerStarted","Data":"6993823c8f99c143f73369a656849011528c5b0b32bcf43b915607a67031f3a2"} Jul 30 14:58:44.029621 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:44.029561 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" podStartSLOduration=1.694516763 podStartE2EDuration="2.029544006s" totalImagesPullingTime="335.027243ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:58:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:58:44.026644075 +0000 UTC m=+651.009927413" watchObservedRunningTime="2026-07-30 14:58:44.029544006 +0000 UTC m=+651.012827343" Jul 30 14:58:53.045123 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:53.045062 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:53.045531 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:53.045151 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:53.057825 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:53.057795 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:58:53.079286 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:58:53.079260 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-5hzdz" Jul 30 14:59:22.733067 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.733027 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755"] Jul 30 14:59:22.736966 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.736947 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.739176 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.739155 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 30 14:59:22.747135 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.747107 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755"] Jul 30 14:59:22.832960 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.832916 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.833369 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.833296 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ttjg\" (UniqueName: \"kubernetes.io/projected/91f68cda-c1d2-455c-8e3a-176eeda73b35-kube-api-access-5ttjg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.833494 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.833411 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.833494 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.833454 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f68cda-c1d2-455c-8e3a-176eeda73b35-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.833494 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.833489 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.833672 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.833547 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935100 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935304 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935112 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935304 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935156 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ttjg\" (UniqueName: \"kubernetes.io/projected/91f68cda-c1d2-455c-8e3a-176eeda73b35-kube-api-access-5ttjg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935304 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935214 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935480 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935326 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f68cda-c1d2-455c-8e3a-176eeda73b35-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935480 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935365 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935637 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935615 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935705 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935681 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.935903 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.935875 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.938266 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.938244 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/91f68cda-c1d2-455c-8e3a-176eeda73b35-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.938424 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.938406 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/91f68cda-c1d2-455c-8e3a-176eeda73b35-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:22.942907 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:22.942884 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ttjg\" (UniqueName: \"kubernetes.io/projected/91f68cda-c1d2-455c-8e3a-176eeda73b35-kube-api-access-5ttjg\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-j9755\" (UID: \"91f68cda-c1d2-455c-8e3a-176eeda73b35\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:23.047817 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:23.047722 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:23.193404 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:23.193374 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755"] Jul 30 14:59:23.196211 ip-10-0-130-38 kubenswrapper[2564]: W0730 14:59:23.196183 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91f68cda_c1d2_455c_8e3a_176eeda73b35.slice/crio-ea5121c1d4e02b4709f4e3fb0f1c96830d91f01bb9913ee4b100d9509f34a03c WatchSource:0}: Error finding container ea5121c1d4e02b4709f4e3fb0f1c96830d91f01bb9913ee4b100d9509f34a03c: Status 404 returned error can't find the container with id ea5121c1d4e02b4709f4e3fb0f1c96830d91f01bb9913ee4b100d9509f34a03c Jul 30 14:59:23.242848 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:23.242818 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" event={"ID":"91f68cda-c1d2-455c-8e3a-176eeda73b35","Type":"ContainerStarted","Data":"ea5121c1d4e02b4709f4e3fb0f1c96830d91f01bb9913ee4b100d9509f34a03c"} Jul 30 14:59:24.250018 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:24.249983 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" event={"ID":"91f68cda-c1d2-455c-8e3a-176eeda73b35","Type":"ContainerStarted","Data":"91d56994f972f8212c566acff0874cf9d7f5288a965e2ea71335b6e205db2f91"} Jul 30 14:59:24.268727 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:24.268674 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" podStartSLOduration=2.062553406 podStartE2EDuration="2.268653805s" totalImagesPullingTime="206.100399ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 14:59:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 14:59:24.267273174 +0000 UTC m=+691.250556512" watchObservedRunningTime="2026-07-30 14:59:24.268653805 +0000 UTC m=+691.251937145" Jul 30 14:59:33.048599 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:33.048562 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:33.049112 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:33.048624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:33.061139 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:33.061115 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 14:59:33.317625 ip-10-0-130-38 kubenswrapper[2564]: I0730 14:59:33.317543 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-j9755" Jul 30 15:01:10.485857 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.485818 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55fb68957-4k4k6"] Jul 30 15:01:10.489461 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.489440 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55fb68957-4k4k6" Jul 30 15:01:10.501409 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.501383 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55fb68957-4k4k6"] Jul 30 15:01:10.605094 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.605049 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a2a5e88-5891-4e44-96cd-86aad43df808-tls-cert\") pod \"authorino-55fb68957-4k4k6\" (UID: \"3a2a5e88-5891-4e44-96cd-86aad43df808\") " pod="kuadrant-system/authorino-55fb68957-4k4k6" Jul 30 15:01:10.605474 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.605444 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9j52\" (UniqueName: \"kubernetes.io/projected/3a2a5e88-5891-4e44-96cd-86aad43df808-kube-api-access-t9j52\") pod \"authorino-55fb68957-4k4k6\" (UID: \"3a2a5e88-5891-4e44-96cd-86aad43df808\") " pod="kuadrant-system/authorino-55fb68957-4k4k6" Jul 30 15:01:10.706941 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.706906 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a2a5e88-5891-4e44-96cd-86aad43df808-tls-cert\") pod \"authorino-55fb68957-4k4k6\" (UID: \"3a2a5e88-5891-4e44-96cd-86aad43df808\") " pod="kuadrant-system/authorino-55fb68957-4k4k6" Jul 30 15:01:10.707121 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.707006 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9j52\" (UniqueName: \"kubernetes.io/projected/3a2a5e88-5891-4e44-96cd-86aad43df808-kube-api-access-t9j52\") pod \"authorino-55fb68957-4k4k6\" (UID: \"3a2a5e88-5891-4e44-96cd-86aad43df808\") " pod="kuadrant-system/authorino-55fb68957-4k4k6" Jul 30 15:01:10.709205 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.709186 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/3a2a5e88-5891-4e44-96cd-86aad43df808-tls-cert\") pod \"authorino-55fb68957-4k4k6\" (UID: \"3a2a5e88-5891-4e44-96cd-86aad43df808\") " pod="kuadrant-system/authorino-55fb68957-4k4k6" Jul 30 15:01:10.714447 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.714427 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9j52\" (UniqueName: \"kubernetes.io/projected/3a2a5e88-5891-4e44-96cd-86aad43df808-kube-api-access-t9j52\") pod \"authorino-55fb68957-4k4k6\" (UID: \"3a2a5e88-5891-4e44-96cd-86aad43df808\") " pod="kuadrant-system/authorino-55fb68957-4k4k6" Jul 30 15:01:10.799519 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.799421 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55fb68957-4k4k6" Jul 30 15:01:10.940833 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:10.940798 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55fb68957-4k4k6"] Jul 30 15:01:10.942179 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:01:10.942152 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a2a5e88_5891_4e44_96cd_86aad43df808.slice/crio-e0aa15292785c90a15c5d8fcce8441f9a75d7dcbb974a52dba5f008cd2e2b76c WatchSource:0}: Error finding container e0aa15292785c90a15c5d8fcce8441f9a75d7dcbb974a52dba5f008cd2e2b76c: Status 404 returned error can't find the container with id e0aa15292785c90a15c5d8fcce8441f9a75d7dcbb974a52dba5f008cd2e2b76c Jul 30 15:01:11.901136 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:11.901021 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55fb68957-4k4k6" event={"ID":"3a2a5e88-5891-4e44-96cd-86aad43df808","Type":"ContainerStarted","Data":"ee6ecdf3d75158beb54daf3488b6b53cf2504b8747b216be51679d260723e6e4"} Jul 30 15:01:11.901136 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:11.901096 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55fb68957-4k4k6" event={"ID":"3a2a5e88-5891-4e44-96cd-86aad43df808","Type":"ContainerStarted","Data":"e0aa15292785c90a15c5d8fcce8441f9a75d7dcbb974a52dba5f008cd2e2b76c"} Jul 30 15:01:11.923872 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:11.923816 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55fb68957-4k4k6" podStartSLOduration=1.287343368 podStartE2EDuration="1.923798s" totalImagesPullingTime="636.454632ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:01:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:01:11.921286847 +0000 UTC m=+798.904570186" watchObservedRunningTime="2026-07-30 15:01:11.923798 +0000 UTC m=+798.907081363" Jul 30 15:01:11.954881 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:11.954845 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5477478cb-k598d"] Jul 30 15:01:11.955101 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:11.955011 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5477478cb-k598d" podUID="e2fcbddc-a73d-4755-ad55-4725b26d7829" containerName="authorino" containerID="cri-o://451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11" gracePeriod=30 Jul 30 15:01:12.241865 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.241833 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 15:01:12.322375 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.322341 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdxwz\" (UniqueName: \"kubernetes.io/projected/e2fcbddc-a73d-4755-ad55-4725b26d7829-kube-api-access-wdxwz\") pod \"e2fcbddc-a73d-4755-ad55-4725b26d7829\" (UID: \"e2fcbddc-a73d-4755-ad55-4725b26d7829\") " Jul 30 15:01:12.322546 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.322493 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2fcbddc-a73d-4755-ad55-4725b26d7829-tls-cert\") pod \"e2fcbddc-a73d-4755-ad55-4725b26d7829\" (UID: \"e2fcbddc-a73d-4755-ad55-4725b26d7829\") " Jul 30 15:01:12.329969 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.329556 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fcbddc-a73d-4755-ad55-4725b26d7829-kube-api-access-wdxwz" (OuterVolumeSpecName: "kube-api-access-wdxwz") pod "e2fcbddc-a73d-4755-ad55-4725b26d7829" (UID: "e2fcbddc-a73d-4755-ad55-4725b26d7829"). InnerVolumeSpecName "kube-api-access-wdxwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:01:12.343849 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.343812 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2fcbddc-a73d-4755-ad55-4725b26d7829-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "e2fcbddc-a73d-4755-ad55-4725b26d7829" (UID: "e2fcbddc-a73d-4755-ad55-4725b26d7829"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:01:12.423736 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.423695 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdxwz\" (UniqueName: \"kubernetes.io/projected/e2fcbddc-a73d-4755-ad55-4725b26d7829-kube-api-access-wdxwz\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:01:12.423736 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.423728 2564 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/e2fcbddc-a73d-4755-ad55-4725b26d7829-tls-cert\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:01:12.908702 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.908667 2564 generic.go:356] "Generic (PLEG): container finished" podID="e2fcbddc-a73d-4755-ad55-4725b26d7829" containerID="451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11" exitCode=0 Jul 30 15:01:12.909218 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.908728 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5477478cb-k598d" Jul 30 15:01:12.909218 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.908764 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5477478cb-k598d" event={"ID":"e2fcbddc-a73d-4755-ad55-4725b26d7829","Type":"ContainerDied","Data":"451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11"} Jul 30 15:01:12.909218 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.908821 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5477478cb-k598d" event={"ID":"e2fcbddc-a73d-4755-ad55-4725b26d7829","Type":"ContainerDied","Data":"1a7d417f778f8a47314b91ad9f70e8e4d2845fb7fa15ebd19a1db51cd9f606f7"} Jul 30 15:01:12.909218 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.908848 2564 scope.go:117] "RemoveContainer" containerID="451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11" Jul 30 15:01:12.919102 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.919054 2564 scope.go:117] "RemoveContainer" containerID="451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11" Jul 30 15:01:12.919488 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:01:12.919459 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11\": container with ID starting with 451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11 not found: ID does not exist" containerID="451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11" Jul 30 15:01:12.919602 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.919500 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11"} err="failed to get container status \"451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11\": rpc error: code = NotFound desc = could not find container \"451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11\": container with ID starting with 451a652c4a696c1fef10c8af6a3065ff6f26d57ee7a41badb4a392f9c3fe7f11 not found: ID does not exist" Jul 30 15:01:12.942679 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.935552 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5477478cb-k598d"] Jul 30 15:01:12.948146 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:12.948106 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5477478cb-k598d"] Jul 30 15:01:13.492528 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:01:13.492491 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2fcbddc-a73d-4755-ad55-4725b26d7829" path="/var/lib/kubelet/pods/e2fcbddc-a73d-4755-ad55-4725b26d7829/volumes" Jul 30 15:02:53.560885 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:02:53.560854 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:02:53.563236 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:02:53.562180 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:07:53.607254 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:07:53.607143 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:07:53.611352 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:07:53.611331 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:10:30.687849 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:30.687760 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg"] Jul 30 15:10:30.688457 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:30.688022 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" podUID="9868e2ce-3164-4d3e-9252-12f4bdeff3fc" containerName="manager" containerID="cri-o://3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6" gracePeriod=10 Jul 30 15:10:30.951393 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:30.951329 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 15:10:31.036678 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.036650 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gvp4v\" (UniqueName: \"kubernetes.io/projected/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-kube-api-access-gvp4v\") pod \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\" (UID: \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\") " Jul 30 15:10:31.036867 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.036703 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-extensions-socket-volume\") pod \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\" (UID: \"9868e2ce-3164-4d3e-9252-12f4bdeff3fc\") " Jul 30 15:10:31.037122 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.037098 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "9868e2ce-3164-4d3e-9252-12f4bdeff3fc" (UID: "9868e2ce-3164-4d3e-9252-12f4bdeff3fc"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:10:31.038674 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.038650 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-kube-api-access-gvp4v" (OuterVolumeSpecName: "kube-api-access-gvp4v") pod "9868e2ce-3164-4d3e-9252-12f4bdeff3fc" (UID: "9868e2ce-3164-4d3e-9252-12f4bdeff3fc"). InnerVolumeSpecName "kube-api-access-gvp4v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:10:31.138065 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.138025 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvp4v\" (UniqueName: \"kubernetes.io/projected/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-kube-api-access-gvp4v\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:10:31.138065 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.138057 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9868e2ce-3164-4d3e-9252-12f4bdeff3fc-extensions-socket-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:10:31.362357 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.362314 2564 generic.go:356] "Generic (PLEG): container finished" podID="9868e2ce-3164-4d3e-9252-12f4bdeff3fc" containerID="3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6" exitCode=0 Jul 30 15:10:31.362544 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.362365 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" Jul 30 15:10:31.362544 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.362405 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" event={"ID":"9868e2ce-3164-4d3e-9252-12f4bdeff3fc","Type":"ContainerDied","Data":"3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6"} Jul 30 15:10:31.362544 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.362489 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg" event={"ID":"9868e2ce-3164-4d3e-9252-12f4bdeff3fc","Type":"ContainerDied","Data":"95b5e17301182ac18d106bfbdfe5410bb9438151036ce5f3e7b0fafff3d086f7"} Jul 30 15:10:31.362544 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.362521 2564 scope.go:117] "RemoveContainer" containerID="3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6" Jul 30 15:10:31.373995 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.373975 2564 scope.go:117] "RemoveContainer" containerID="3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6" Jul 30 15:10:31.374353 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:10:31.374331 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6\": container with ID starting with 3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6 not found: ID does not exist" containerID="3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6" Jul 30 15:10:31.374411 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.374362 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6"} err="failed to get container status \"3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6\": rpc error: code = NotFound desc = could not find container \"3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6\": container with ID starting with 3504a4933ac93dbbaf98d24a954edb4b0788536f46a3868b3a05af7038c25bd6 not found: ID does not exist" Jul 30 15:10:31.383719 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.383695 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg"] Jul 30 15:10:31.387624 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.387595 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-4cbsg"] Jul 30 15:10:31.490998 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:10:31.490956 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9868e2ce-3164-4d3e-9252-12f4bdeff3fc" path="/var/lib/kubelet/pods/9868e2ce-3164-4d3e-9252-12f4bdeff3fc/volumes" Jul 30 15:11:31.491333 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.491297 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622"] Jul 30 15:11:31.491816 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.491737 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fcbddc-a73d-4755-ad55-4725b26d7829" containerName="authorino" Jul 30 15:11:31.491816 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.491748 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fcbddc-a73d-4755-ad55-4725b26d7829" containerName="authorino" Jul 30 15:11:31.491816 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.491771 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9868e2ce-3164-4d3e-9252-12f4bdeff3fc" containerName="manager" Jul 30 15:11:31.491816 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.491777 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="9868e2ce-3164-4d3e-9252-12f4bdeff3fc" containerName="manager" Jul 30 15:11:31.491949 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.491861 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="9868e2ce-3164-4d3e-9252-12f4bdeff3fc" containerName="manager" Jul 30 15:11:31.491949 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.491877 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2fcbddc-a73d-4755-ad55-4725b26d7829" containerName="authorino" Jul 30 15:11:31.494985 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.494969 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:31.497344 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.497323 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4kdz\"" Jul 30 15:11:31.499914 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.499895 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622"] Jul 30 15:11:31.514543 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.514512 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpsfl\" (UniqueName: \"kubernetes.io/projected/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-kube-api-access-jpsfl\") pod \"kuadrant-operator-controller-manager-86878c769f-rn622\" (UID: \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:31.514664 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.514557 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rn622\" (UID: \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:31.615914 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.615878 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpsfl\" (UniqueName: \"kubernetes.io/projected/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-kube-api-access-jpsfl\") pod \"kuadrant-operator-controller-manager-86878c769f-rn622\" (UID: \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:31.615914 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.615918 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rn622\" (UID: \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:31.616267 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.616251 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-rn622\" (UID: \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:31.624704 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.624681 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpsfl\" (UniqueName: \"kubernetes.io/projected/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-kube-api-access-jpsfl\") pod \"kuadrant-operator-controller-manager-86878c769f-rn622\" (UID: \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:31.807205 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.807112 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:31.952971 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:31.952947 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622"] Jul 30 15:11:31.954731 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:11:31.954703 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd59c7fe3_1fc2_408c_8524_bbf58612c5c9.slice/crio-c8e25a9e679a55604e54d4c9f11b6fe97cb91b64dae65cec50220e7cc84c6006 WatchSource:0}: Error finding container c8e25a9e679a55604e54d4c9f11b6fe97cb91b64dae65cec50220e7cc84c6006: Status 404 returned error can't find the container with id c8e25a9e679a55604e54d4c9f11b6fe97cb91b64dae65cec50220e7cc84c6006 Jul 30 15:11:32.721748 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:32.721712 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" event={"ID":"d59c7fe3-1fc2-408c-8524-bbf58612c5c9","Type":"ContainerStarted","Data":"5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29"} Jul 30 15:11:32.722227 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:32.721789 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" event={"ID":"d59c7fe3-1fc2-408c-8524-bbf58612c5c9","Type":"ContainerStarted","Data":"c8e25a9e679a55604e54d4c9f11b6fe97cb91b64dae65cec50220e7cc84c6006"} Jul 30 15:11:32.722227 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:32.721809 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:11:32.744196 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:32.744148 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" podStartSLOduration=1.744133165 podStartE2EDuration="1.744133165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:11:32.742446069 +0000 UTC m=+1419.725729441" watchObservedRunningTime="2026-07-30 15:11:32.744133165 +0000 UTC m=+1419.727416519" Jul 30 15:11:43.730224 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:11:43.730188 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:12:53.653173 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:12:53.653042 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:12:53.659047 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:12:53.659024 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:17:53.694865 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:17:53.694754 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:17:53.703411 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:17:53.703387 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:21:08.983776 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:08.983695 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5"] Jul 30 15:21:08.987307 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:08.987289 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:08.989227 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:08.989205 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-624a770e\"" Jul 30 15:21:08.989369 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:08.989248 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-624a770e-dockercfg-zvkhx\"" Jul 30 15:21:08.997187 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:08.997161 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5"] Jul 30 15:21:09.049587 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:09.049553 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5\" (UID: \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:09.049784 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:09.049666 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dwkr\" (UniqueName: \"kubernetes.io/projected/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-kube-api-access-8dwkr\") pod \"payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5\" (UID: \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:09.150418 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:09.150385 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5\" (UID: \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:09.150591 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:09.150452 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dwkr\" (UniqueName: \"kubernetes.io/projected/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-kube-api-access-8dwkr\") pod \"payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5\" (UID: \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:09.151146 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:09.151126 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5\" (UID: \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:09.162100 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:09.162059 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dwkr\" (UniqueName: \"kubernetes.io/projected/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-kube-api-access-8dwkr\") pod \"payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5\" (UID: \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:09.299817 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:09.299716 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:09.443263 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:09.443237 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5"] Jul 30 15:21:09.445688 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:21:09.445659 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07fc50f8_05c5_411c_afa4_f7a42f2e1ec5.slice/crio-68a7d10584a06ad965e414447955336fc0563a3e90206d0364c22f4886082aad WatchSource:0}: Error finding container 68a7d10584a06ad965e414447955336fc0563a3e90206d0364c22f4886082aad: Status 404 returned error can't find the container with id 68a7d10584a06ad965e414447955336fc0563a3e90206d0364c22f4886082aad Jul 30 15:21:10.291743 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:10.291706 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" event={"ID":"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5","Type":"ContainerStarted","Data":"fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0"} Jul 30 15:21:10.292170 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:10.291918 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:10.292170 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:10.291958 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" event={"ID":"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5","Type":"ContainerStarted","Data":"68a7d10584a06ad965e414447955336fc0563a3e90206d0364c22f4886082aad"} Jul 30 15:21:10.310445 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:10.310392 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" podStartSLOduration=2.310374666 podStartE2EDuration="2.310374666s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:21:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:21:10.309064828 +0000 UTC m=+1997.292348166" watchObservedRunningTime="2026-07-30 15:21:10.310374666 +0000 UTC m=+1997.293658003" Jul 30 15:21:21.299626 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:21.299597 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:48.870686 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:48.870650 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5"] Jul 30 15:21:48.871300 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:48.870908 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" podUID="07fc50f8-05c5-411c-afa4-f7a42f2e1ec5" containerName="payload-pre-processing" containerID="cri-o://fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0" gracePeriod=30 Jul 30 15:21:49.135955 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.135872 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:49.237279 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.237239 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-plugins-config-volume\") pod \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\" (UID: \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\") " Jul 30 15:21:49.237478 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.237358 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dwkr\" (UniqueName: \"kubernetes.io/projected/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-kube-api-access-8dwkr\") pod \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\" (UID: \"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5\") " Jul 30 15:21:49.237677 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.237649 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "07fc50f8-05c5-411c-afa4-f7a42f2e1ec5" (UID: "07fc50f8-05c5-411c-afa4-f7a42f2e1ec5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:21:49.239648 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.239618 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-kube-api-access-8dwkr" (OuterVolumeSpecName: "kube-api-access-8dwkr") pod "07fc50f8-05c5-411c-afa4-f7a42f2e1ec5" (UID: "07fc50f8-05c5-411c-afa4-f7a42f2e1ec5"). InnerVolumeSpecName "kube-api-access-8dwkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:21:49.338715 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.338678 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dwkr\" (UniqueName: \"kubernetes.io/projected/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-kube-api-access-8dwkr\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:21:49.338715 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.338707 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5-plugins-config-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:21:49.528215 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.528186 2564 generic.go:356] "Generic (PLEG): container finished" podID="07fc50f8-05c5-411c-afa4-f7a42f2e1ec5" containerID="fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0" exitCode=0 Jul 30 15:21:49.528361 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.528228 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" Jul 30 15:21:49.528361 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.528257 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" event={"ID":"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5","Type":"ContainerDied","Data":"fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0"} Jul 30 15:21:49.528361 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.528296 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5" event={"ID":"07fc50f8-05c5-411c-afa4-f7a42f2e1ec5","Type":"ContainerDied","Data":"68a7d10584a06ad965e414447955336fc0563a3e90206d0364c22f4886082aad"} Jul 30 15:21:49.528361 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.528317 2564 scope.go:117] "RemoveContainer" containerID="fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0" Jul 30 15:21:49.537546 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.537528 2564 scope.go:117] "RemoveContainer" containerID="fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0" Jul 30 15:21:49.537834 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:21:49.537813 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0\": container with ID starting with fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0 not found: ID does not exist" containerID="fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0" Jul 30 15:21:49.537883 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.537848 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0"} err="failed to get container status \"fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0\": rpc error: code = NotFound desc = could not find container \"fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0\": container with ID starting with fc9d77e9a1d929efd6dac8681b1bda29a3ef4b17a7917e75061dd342930e87a0 not found: ID does not exist" Jul 30 15:21:49.569589 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.569556 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5"] Jul 30 15:21:49.577918 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:49.577882 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-624a770e-57f7ff69bd-z2wt5"] Jul 30 15:21:51.491459 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:51.491426 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07fc50f8-05c5-411c-afa4-f7a42f2e1ec5" path="/var/lib/kubelet/pods/07fc50f8-05c5-411c-afa4-f7a42f2e1ec5/volumes" Jul 30 15:21:53.451146 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.451110 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc"] Jul 30 15:21:53.451595 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.451583 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07fc50f8-05c5-411c-afa4-f7a42f2e1ec5" containerName="payload-pre-processing" Jul 30 15:21:53.451595 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.451596 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="07fc50f8-05c5-411c-afa4-f7a42f2e1ec5" containerName="payload-pre-processing" Jul 30 15:21:53.451674 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.451664 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="07fc50f8-05c5-411c-afa4-f7a42f2e1ec5" containerName="payload-pre-processing" Jul 30 15:21:53.456251 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.456233 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:53.458098 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.458065 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-5d54eb26-dockercfg-dshs4\"" Jul 30 15:21:53.458253 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.458144 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-5d54eb26\"" Jul 30 15:21:53.465894 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.465871 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc"] Jul 30 15:21:53.479659 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.479628 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ss9s\" (UniqueName: \"kubernetes.io/projected/5cc68512-f246-41db-b708-19f455a9e601-kube-api-access-9ss9s\") pod \"payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc\" (UID: \"5cc68512-f246-41db-b708-19f455a9e601\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:53.479824 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.479755 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc68512-f246-41db-b708-19f455a9e601-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc\" (UID: \"5cc68512-f246-41db-b708-19f455a9e601\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:53.581097 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.581035 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc68512-f246-41db-b708-19f455a9e601-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc\" (UID: \"5cc68512-f246-41db-b708-19f455a9e601\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:53.581277 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.581124 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ss9s\" (UniqueName: \"kubernetes.io/projected/5cc68512-f246-41db-b708-19f455a9e601-kube-api-access-9ss9s\") pod \"payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc\" (UID: \"5cc68512-f246-41db-b708-19f455a9e601\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:53.581912 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.581890 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc68512-f246-41db-b708-19f455a9e601-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc\" (UID: \"5cc68512-f246-41db-b708-19f455a9e601\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:53.588649 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.588620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ss9s\" (UniqueName: \"kubernetes.io/projected/5cc68512-f246-41db-b708-19f455a9e601-kube-api-access-9ss9s\") pod \"payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc\" (UID: \"5cc68512-f246-41db-b708-19f455a9e601\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:53.768841 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.768795 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:53.916170 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:53.916147 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc"] Jul 30 15:21:53.918885 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:21:53.918854 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cc68512_f246_41db_b708_19f455a9e601.slice/crio-76be2377c1e6ea539aee2854924564483646622974e093d6db63659e70d80b1f WatchSource:0}: Error finding container 76be2377c1e6ea539aee2854924564483646622974e093d6db63659e70d80b1f: Status 404 returned error can't find the container with id 76be2377c1e6ea539aee2854924564483646622974e093d6db63659e70d80b1f Jul 30 15:21:54.567827 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:54.567786 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" event={"ID":"5cc68512-f246-41db-b708-19f455a9e601","Type":"ContainerStarted","Data":"3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52"} Jul 30 15:21:54.568326 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:54.567853 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" event={"ID":"5cc68512-f246-41db-b708-19f455a9e601","Type":"ContainerStarted","Data":"76be2377c1e6ea539aee2854924564483646622974e093d6db63659e70d80b1f"} Jul 30 15:21:54.568326 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:54.567916 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:21:54.584363 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:21:54.584318 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" podStartSLOduration=1.584302632 podStartE2EDuration="1.584302632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:21:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:21:54.581519201 +0000 UTC m=+2041.564802556" watchObservedRunningTime="2026-07-30 15:21:54.584302632 +0000 UTC m=+2041.567585969" Jul 30 15:22:05.576829 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:05.576794 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:22:24.040905 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.040719 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc"] Jul 30 15:22:24.041519 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.040985 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" podUID="5cc68512-f246-41db-b708-19f455a9e601" containerName="payload-pre-processing" containerID="cri-o://3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52" gracePeriod=30 Jul 30 15:22:24.341664 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.341634 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:22:24.397319 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.397283 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc68512-f246-41db-b708-19f455a9e601-plugins-config-volume\") pod \"5cc68512-f246-41db-b708-19f455a9e601\" (UID: \"5cc68512-f246-41db-b708-19f455a9e601\") " Jul 30 15:22:24.397490 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.397347 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9ss9s\" (UniqueName: \"kubernetes.io/projected/5cc68512-f246-41db-b708-19f455a9e601-kube-api-access-9ss9s\") pod \"5cc68512-f246-41db-b708-19f455a9e601\" (UID: \"5cc68512-f246-41db-b708-19f455a9e601\") " Jul 30 15:22:24.397656 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.397632 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cc68512-f246-41db-b708-19f455a9e601-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5cc68512-f246-41db-b708-19f455a9e601" (UID: "5cc68512-f246-41db-b708-19f455a9e601"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:22:24.399461 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.399428 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cc68512-f246-41db-b708-19f455a9e601-kube-api-access-9ss9s" (OuterVolumeSpecName: "kube-api-access-9ss9s") pod "5cc68512-f246-41db-b708-19f455a9e601" (UID: "5cc68512-f246-41db-b708-19f455a9e601"). InnerVolumeSpecName "kube-api-access-9ss9s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:22:24.498410 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.498365 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ss9s\" (UniqueName: \"kubernetes.io/projected/5cc68512-f246-41db-b708-19f455a9e601-kube-api-access-9ss9s\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:22:24.498410 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.498397 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cc68512-f246-41db-b708-19f455a9e601-plugins-config-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:22:24.754532 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.754499 2564 generic.go:356] "Generic (PLEG): container finished" podID="5cc68512-f246-41db-b708-19f455a9e601" containerID="3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52" exitCode=0 Jul 30 15:22:24.754714 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.754550 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" Jul 30 15:22:24.754714 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.754571 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" event={"ID":"5cc68512-f246-41db-b708-19f455a9e601","Type":"ContainerDied","Data":"3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52"} Jul 30 15:22:24.754714 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.754614 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc" event={"ID":"5cc68512-f246-41db-b708-19f455a9e601","Type":"ContainerDied","Data":"76be2377c1e6ea539aee2854924564483646622974e093d6db63659e70d80b1f"} Jul 30 15:22:24.754714 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.754635 2564 scope.go:117] "RemoveContainer" containerID="3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52" Jul 30 15:22:24.764173 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.764152 2564 scope.go:117] "RemoveContainer" containerID="3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52" Jul 30 15:22:24.765264 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:22:24.765233 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52\": container with ID starting with 3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52 not found: ID does not exist" containerID="3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52" Jul 30 15:22:24.765384 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.765274 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52"} err="failed to get container status \"3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52\": rpc error: code = NotFound desc = could not find container \"3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52\": container with ID starting with 3648a94ec7271a0d565be0cead68e0ed84ab2717d9bb3b2ecb222f563c590f52 not found: ID does not exist" Jul 30 15:22:24.774909 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.774883 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc"] Jul 30 15:22:24.778215 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:24.778193 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-5d54eb26-7d95c4d4fb-s56sc"] Jul 30 15:22:25.490859 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:25.490815 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cc68512-f246-41db-b708-19f455a9e601" path="/var/lib/kubelet/pods/5cc68512-f246-41db-b708-19f455a9e601/volumes" Jul 30 15:22:29.519486 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.519448 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5"] Jul 30 15:22:29.519963 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.519949 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cc68512-f246-41db-b708-19f455a9e601" containerName="payload-pre-processing" Jul 30 15:22:29.520011 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.519965 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cc68512-f246-41db-b708-19f455a9e601" containerName="payload-pre-processing" Jul 30 15:22:29.520062 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.520052 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cc68512-f246-41db-b708-19f455a9e601" containerName="payload-pre-processing" Jul 30 15:22:29.523264 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.523247 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:29.525347 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.525325 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-73521211-dockercfg-87bm2\"" Jul 30 15:22:29.525452 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.525325 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-73521211\"" Jul 30 15:22:29.532731 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.532694 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5"] Jul 30 15:22:29.538790 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.538762 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a123ad80-a8b0-4e3b-b861-0e4234ef060d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5\" (UID: \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:29.538948 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.538926 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4tvq\" (UniqueName: \"kubernetes.io/projected/a123ad80-a8b0-4e3b-b861-0e4234ef060d-kube-api-access-j4tvq\") pod \"payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5\" (UID: \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:29.639995 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.639959 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4tvq\" (UniqueName: \"kubernetes.io/projected/a123ad80-a8b0-4e3b-b861-0e4234ef060d-kube-api-access-j4tvq\") pod \"payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5\" (UID: \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:29.640184 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.640049 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a123ad80-a8b0-4e3b-b861-0e4234ef060d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5\" (UID: \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:29.641017 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.640992 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a123ad80-a8b0-4e3b-b861-0e4234ef060d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5\" (UID: \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:29.649174 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.649140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4tvq\" (UniqueName: \"kubernetes.io/projected/a123ad80-a8b0-4e3b-b861-0e4234ef060d-kube-api-access-j4tvq\") pod \"payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5\" (UID: \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:29.834945 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.834850 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:29.984883 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:29.984857 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5"] Jul 30 15:22:29.987107 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:22:29.987044 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda123ad80_a8b0_4e3b_b861_0e4234ef060d.slice/crio-7633af7984eaee3d60ed5782ddb87f72108f60593b01888ca535b0087047df6c WatchSource:0}: Error finding container 7633af7984eaee3d60ed5782ddb87f72108f60593b01888ca535b0087047df6c: Status 404 returned error can't find the container with id 7633af7984eaee3d60ed5782ddb87f72108f60593b01888ca535b0087047df6c Jul 30 15:22:30.795471 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:30.795437 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" event={"ID":"a123ad80-a8b0-4e3b-b861-0e4234ef060d","Type":"ContainerStarted","Data":"a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85"} Jul 30 15:22:30.795965 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:30.795500 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" event={"ID":"a123ad80-a8b0-4e3b-b861-0e4234ef060d","Type":"ContainerStarted","Data":"7633af7984eaee3d60ed5782ddb87f72108f60593b01888ca535b0087047df6c"} Jul 30 15:22:30.795965 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:30.795618 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:30.813304 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:30.813253 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" podStartSLOduration=1.813236622 podStartE2EDuration="1.813236622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:22:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:22:30.811189397 +0000 UTC m=+2077.794472729" watchObservedRunningTime="2026-07-30 15:22:30.813236622 +0000 UTC m=+2077.796519986" Jul 30 15:22:41.805103 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:41.805050 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:53.736330 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:53.736292 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:22:53.747833 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:53.747810 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:22:59.462616 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.462563 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5"] Jul 30 15:22:59.463288 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.462824 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" podUID="a123ad80-a8b0-4e3b-b861-0e4234ef060d" containerName="payload-pre-processing" containerID="cri-o://a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85" gracePeriod=30 Jul 30 15:22:59.732820 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.732790 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:59.845258 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.845213 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a123ad80-a8b0-4e3b-b861-0e4234ef060d-plugins-config-volume\") pod \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\" (UID: \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\") " Jul 30 15:22:59.845465 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.845277 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4tvq\" (UniqueName: \"kubernetes.io/projected/a123ad80-a8b0-4e3b-b861-0e4234ef060d-kube-api-access-j4tvq\") pod \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\" (UID: \"a123ad80-a8b0-4e3b-b861-0e4234ef060d\") " Jul 30 15:22:59.845683 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.845657 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a123ad80-a8b0-4e3b-b861-0e4234ef060d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a123ad80-a8b0-4e3b-b861-0e4234ef060d" (UID: "a123ad80-a8b0-4e3b-b861-0e4234ef060d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:22:59.847432 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.847408 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a123ad80-a8b0-4e3b-b861-0e4234ef060d-kube-api-access-j4tvq" (OuterVolumeSpecName: "kube-api-access-j4tvq") pod "a123ad80-a8b0-4e3b-b861-0e4234ef060d" (UID: "a123ad80-a8b0-4e3b-b861-0e4234ef060d"). InnerVolumeSpecName "kube-api-access-j4tvq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:22:59.946379 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.946340 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a123ad80-a8b0-4e3b-b861-0e4234ef060d-plugins-config-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:22:59.946379 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.946370 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4tvq\" (UniqueName: \"kubernetes.io/projected/a123ad80-a8b0-4e3b-b861-0e4234ef060d-kube-api-access-j4tvq\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:22:59.978767 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.978738 2564 generic.go:356] "Generic (PLEG): container finished" podID="a123ad80-a8b0-4e3b-b861-0e4234ef060d" containerID="a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85" exitCode=0 Jul 30 15:22:59.978931 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.978787 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" Jul 30 15:22:59.978931 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.978830 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" event={"ID":"a123ad80-a8b0-4e3b-b861-0e4234ef060d","Type":"ContainerDied","Data":"a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85"} Jul 30 15:22:59.978931 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.978889 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5" event={"ID":"a123ad80-a8b0-4e3b-b861-0e4234ef060d","Type":"ContainerDied","Data":"7633af7984eaee3d60ed5782ddb87f72108f60593b01888ca535b0087047df6c"} Jul 30 15:22:59.978931 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.978917 2564 scope.go:117] "RemoveContainer" containerID="a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85" Jul 30 15:22:59.994090 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.994055 2564 scope.go:117] "RemoveContainer" containerID="a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85" Jul 30 15:22:59.994413 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:22:59.994379 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85\": container with ID starting with a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85 not found: ID does not exist" containerID="a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85" Jul 30 15:22:59.994523 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:22:59.994416 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85"} err="failed to get container status \"a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85\": rpc error: code = NotFound desc = could not find container \"a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85\": container with ID starting with a89c313be32ef35d54762065f15f8c1fcd77e795bef81fc253209a668d73aa85 not found: ID does not exist" Jul 30 15:23:00.002884 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:23:00.002860 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5"] Jul 30 15:23:00.006358 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:23:00.006333 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-73521211-6b8d8f8984-nctk5"] Jul 30 15:23:01.491815 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:23:01.491776 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a123ad80-a8b0-4e3b-b861-0e4234ef060d" path="/var/lib/kubelet/pods/a123ad80-a8b0-4e3b-b861-0e4234ef060d/volumes" Jul 30 15:27:21.488363 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:21.488321 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622"] Jul 30 15:27:21.489836 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:21.489777 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" podUID="d59c7fe3-1fc2-408c-8524-bbf58612c5c9" containerName="manager" containerID="cri-o://5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29" gracePeriod=10 Jul 30 15:27:23.353417 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.353388 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:27:23.434488 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.434453 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-extensions-socket-volume\") pod \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\" (UID: \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\") " Jul 30 15:27:23.434663 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.434567 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpsfl\" (UniqueName: \"kubernetes.io/projected/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-kube-api-access-jpsfl\") pod \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\" (UID: \"d59c7fe3-1fc2-408c-8524-bbf58612c5c9\") " Jul 30 15:27:23.434932 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.434901 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d59c7fe3-1fc2-408c-8524-bbf58612c5c9" (UID: "d59c7fe3-1fc2-408c-8524-bbf58612c5c9"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:27:23.436584 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.436555 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-kube-api-access-jpsfl" (OuterVolumeSpecName: "kube-api-access-jpsfl") pod "d59c7fe3-1fc2-408c-8524-bbf58612c5c9" (UID: "d59c7fe3-1fc2-408c-8524-bbf58612c5c9"). InnerVolumeSpecName "kube-api-access-jpsfl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:27:23.535486 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.535451 2564 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-extensions-socket-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:27:23.535486 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.535480 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpsfl\" (UniqueName: \"kubernetes.io/projected/d59c7fe3-1fc2-408c-8524-bbf58612c5c9-kube-api-access-jpsfl\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:27:23.639046 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.639005 2564 generic.go:356] "Generic (PLEG): container finished" podID="d59c7fe3-1fc2-408c-8524-bbf58612c5c9" containerID="5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29" exitCode=0 Jul 30 15:27:23.639223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.639053 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" Jul 30 15:27:23.639223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.639115 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" event={"ID":"d59c7fe3-1fc2-408c-8524-bbf58612c5c9","Type":"ContainerDied","Data":"5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29"} Jul 30 15:27:23.639223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.639177 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622" event={"ID":"d59c7fe3-1fc2-408c-8524-bbf58612c5c9","Type":"ContainerDied","Data":"c8e25a9e679a55604e54d4c9f11b6fe97cb91b64dae65cec50220e7cc84c6006"} Jul 30 15:27:23.639223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.639199 2564 scope.go:117] "RemoveContainer" containerID="5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29" Jul 30 15:27:23.649992 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.649969 2564 scope.go:117] "RemoveContainer" containerID="5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29" Jul 30 15:27:23.650273 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:27:23.650253 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29\": container with ID starting with 5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29 not found: ID does not exist" containerID="5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29" Jul 30 15:27:23.650343 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.650282 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29"} err="failed to get container status \"5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29\": rpc error: code = NotFound desc = could not find container \"5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29\": container with ID starting with 5533205a6779f12dd346e12d4a08387697610af4b14364f285a5d02215962b29 not found: ID does not exist" Jul 30 15:27:23.656678 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.656655 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622"] Jul 30 15:27:23.662163 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:23.662139 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-rn622"] Jul 30 15:27:25.491424 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:25.491390 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d59c7fe3-1fc2-408c-8524-bbf58612c5c9" path="/var/lib/kubelet/pods/d59c7fe3-1fc2-408c-8524-bbf58612c5c9/volumes" Jul 30 15:27:53.777644 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:53.777525 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:27:53.790815 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:27:53.790793 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:28:37.140428 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.140345 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52"] Jul 30 15:28:37.140905 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.140799 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a123ad80-a8b0-4e3b-b861-0e4234ef060d" containerName="payload-pre-processing" Jul 30 15:28:37.140905 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.140810 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="a123ad80-a8b0-4e3b-b861-0e4234ef060d" containerName="payload-pre-processing" Jul 30 15:28:37.140905 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.140834 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d59c7fe3-1fc2-408c-8524-bbf58612c5c9" containerName="manager" Jul 30 15:28:37.140905 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.140839 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="d59c7fe3-1fc2-408c-8524-bbf58612c5c9" containerName="manager" Jul 30 15:28:37.141043 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.140909 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="a123ad80-a8b0-4e3b-b861-0e4234ef060d" containerName="payload-pre-processing" Jul 30 15:28:37.141043 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.140918 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="d59c7fe3-1fc2-408c-8524-bbf58612c5c9" containerName="manager" Jul 30 15:28:37.144170 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.144151 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:37.146427 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.146402 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-n4kdz\"" Jul 30 15:28:37.155635 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.155612 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52"] Jul 30 15:28:37.271596 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.271555 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/207dd104-3c82-46b5-886d-4036d4c0cb6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dxc52\" (UID: \"207dd104-3c82-46b5-886d-4036d4c0cb6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:37.271784 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.271613 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xwjn\" (UniqueName: \"kubernetes.io/projected/207dd104-3c82-46b5-886d-4036d4c0cb6c-kube-api-access-6xwjn\") pod \"kuadrant-operator-controller-manager-86878c769f-dxc52\" (UID: \"207dd104-3c82-46b5-886d-4036d4c0cb6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:37.373024 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.372982 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/207dd104-3c82-46b5-886d-4036d4c0cb6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dxc52\" (UID: \"207dd104-3c82-46b5-886d-4036d4c0cb6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:37.373226 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.373044 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xwjn\" (UniqueName: \"kubernetes.io/projected/207dd104-3c82-46b5-886d-4036d4c0cb6c-kube-api-access-6xwjn\") pod \"kuadrant-operator-controller-manager-86878c769f-dxc52\" (UID: \"207dd104-3c82-46b5-886d-4036d4c0cb6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:37.373409 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.373387 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/207dd104-3c82-46b5-886d-4036d4c0cb6c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-dxc52\" (UID: \"207dd104-3c82-46b5-886d-4036d4c0cb6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:37.381130 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.381103 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xwjn\" (UniqueName: \"kubernetes.io/projected/207dd104-3c82-46b5-886d-4036d4c0cb6c-kube-api-access-6xwjn\") pod \"kuadrant-operator-controller-manager-86878c769f-dxc52\" (UID: \"207dd104-3c82-46b5-886d-4036d4c0cb6c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:37.455038 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.454944 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:37.604067 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:37.604042 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52"] Jul 30 15:28:37.607009 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:28:37.606979 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod207dd104_3c82_46b5_886d_4036d4c0cb6c.slice/crio-7d633f56c63a00e892cd0ee8ab00759ad08318734657c734d743a9cbdecb8182 WatchSource:0}: Error finding container 7d633f56c63a00e892cd0ee8ab00759ad08318734657c734d743a9cbdecb8182: Status 404 returned error can't find the container with id 7d633f56c63a00e892cd0ee8ab00759ad08318734657c734d743a9cbdecb8182 Jul 30 15:28:38.085771 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:38.085721 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" event={"ID":"207dd104-3c82-46b5-886d-4036d4c0cb6c","Type":"ContainerStarted","Data":"61045830d94b2049e44518562898d52a98cd0aad9173af7f86b0359250b256ea"} Jul 30 15:28:38.085994 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:38.085804 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:28:38.085994 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:38.085828 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" event={"ID":"207dd104-3c82-46b5-886d-4036d4c0cb6c","Type":"ContainerStarted","Data":"7d633f56c63a00e892cd0ee8ab00759ad08318734657c734d743a9cbdecb8182"} Jul 30 15:28:38.107221 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:38.107167 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" podStartSLOduration=1.107151012 podStartE2EDuration="1.107151012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:28:38.104740594 +0000 UTC m=+2445.088023931" watchObservedRunningTime="2026-07-30 15:28:38.107151012 +0000 UTC m=+2445.090434349" Jul 30 15:28:49.093800 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:28:49.093767 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-dxc52" Jul 30 15:29:44.627578 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.627538 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8"] Jul 30 15:29:44.631412 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.631392 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.633504 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.633483 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2fa717fb\"/\"openshift-service-ca.crt\"" Jul 30 15:29:44.633921 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.633884 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2fa717fb\"/\"e2e-lifecycle-model-2fa717fb-kserve-self-signed-certs\"" Jul 30 15:29:44.633921 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.633908 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-2fa717fb\"/\"default-dockercfg-w7tk5\"" Jul 30 15:29:44.634140 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.633929 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-2fa717fb\"/\"kube-root-ca.crt\"" Jul 30 15:29:44.641552 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.641524 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8"] Jul 30 15:29:44.737722 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.737676 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-model-cache\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.737894 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.737750 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-dshm\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.737894 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.737775 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-tmp-dir\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.737894 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.737809 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-home\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.737894 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.737828 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jfghg\" (UniqueName: \"kubernetes.io/projected/f873e236-8ffb-4112-8dcf-2d117a5841dc-kube-api-access-jfghg\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.738122 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.737921 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873e236-8ffb-4112-8dcf-2d117a5841dc-tls-certs\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.838544 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.838472 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-model-cache\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.838544 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.838543 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-dshm\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.838812 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.838567 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-tmp-dir\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.838812 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.838602 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-home\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.838812 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.838620 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jfghg\" (UniqueName: \"kubernetes.io/projected/f873e236-8ffb-4112-8dcf-2d117a5841dc-kube-api-access-jfghg\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.838812 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.838664 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873e236-8ffb-4112-8dcf-2d117a5841dc-tls-certs\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.839161 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.838974 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-model-cache\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.839161 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.839049 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-tmp-dir\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.839493 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.839177 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-home\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.841012 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.840987 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-dshm\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.841512 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.841491 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873e236-8ffb-4112-8dcf-2d117a5841dc-tls-certs\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.848149 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.847386 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jfghg\" (UniqueName: \"kubernetes.io/projected/f873e236-8ffb-4112-8dcf-2d117a5841dc-kube-api-access-jfghg\") pod \"e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:44.941370 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:44.941282 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:45.096749 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:45.096721 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8"] Jul 30 15:29:45.098732 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:29:45.098704 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf873e236_8ffb_4112_8dcf_2d117a5841dc.slice/crio-6ad36a0c83b576b909be5a6c58621df7b37199371bb59baedd7f573a008caafb WatchSource:0}: Error finding container 6ad36a0c83b576b909be5a6c58621df7b37199371bb59baedd7f573a008caafb: Status 404 returned error can't find the container with id 6ad36a0c83b576b909be5a6c58621df7b37199371bb59baedd7f573a008caafb Jul 30 15:29:45.501491 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:45.501450 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" event={"ID":"f873e236-8ffb-4112-8dcf-2d117a5841dc","Type":"ContainerStarted","Data":"be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d"} Jul 30 15:29:45.501491 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:45.501497 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" event={"ID":"f873e236-8ffb-4112-8dcf-2d117a5841dc","Type":"ContainerStarted","Data":"6ad36a0c83b576b909be5a6c58621df7b37199371bb59baedd7f573a008caafb"} Jul 30 15:29:45.522785 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:45.522729 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" podStartSLOduration=1.5227102430000001 podStartE2EDuration="1.522710243s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:29:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:29:45.521825707 +0000 UTC m=+2512.505109035" watchObservedRunningTime="2026-07-30 15:29:45.522710243 +0000 UTC m=+2512.505993580" Jul 30 15:29:54.942318 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:54.942278 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:54.942318 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:54.942326 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:54.954893 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:54.954862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:29:55.574894 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:29:55.574857 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:30:43.816350 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:43.816309 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8"] Jul 30 15:30:58.836687 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:58.836643 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" podUID="f873e236-8ffb-4112-8dcf-2d117a5841dc" containerName="main" containerID="cri-o://be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d" gracePeriod=45 Jul 30 15:30:59.104360 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.104292 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:30:59.187435 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.187405 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873e236-8ffb-4112-8dcf-2d117a5841dc-tls-certs\") pod \"f873e236-8ffb-4112-8dcf-2d117a5841dc\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " Jul 30 15:30:59.187435 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.187446 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-home\") pod \"f873e236-8ffb-4112-8dcf-2d117a5841dc\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " Jul 30 15:30:59.187710 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.187589 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-dshm\") pod \"f873e236-8ffb-4112-8dcf-2d117a5841dc\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " Jul 30 15:30:59.187710 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.187621 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jfghg\" (UniqueName: \"kubernetes.io/projected/f873e236-8ffb-4112-8dcf-2d117a5841dc-kube-api-access-jfghg\") pod \"f873e236-8ffb-4112-8dcf-2d117a5841dc\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " Jul 30 15:30:59.187710 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.187671 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-tmp-dir\") pod \"f873e236-8ffb-4112-8dcf-2d117a5841dc\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " Jul 30 15:30:59.187710 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.187699 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-model-cache\") pod \"f873e236-8ffb-4112-8dcf-2d117a5841dc\" (UID: \"f873e236-8ffb-4112-8dcf-2d117a5841dc\") " Jul 30 15:30:59.187920 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.187758 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-home" (OuterVolumeSpecName: "home") pod "f873e236-8ffb-4112-8dcf-2d117a5841dc" (UID: "f873e236-8ffb-4112-8dcf-2d117a5841dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:30:59.188029 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.187975 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f873e236-8ffb-4112-8dcf-2d117a5841dc" (UID: "f873e236-8ffb-4112-8dcf-2d117a5841dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:30:59.188117 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.188010 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-home\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:30:59.188117 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.188040 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "f873e236-8ffb-4112-8dcf-2d117a5841dc" (UID: "f873e236-8ffb-4112-8dcf-2d117a5841dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:30:59.189736 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.189710 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f873e236-8ffb-4112-8dcf-2d117a5841dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f873e236-8ffb-4112-8dcf-2d117a5841dc" (UID: "f873e236-8ffb-4112-8dcf-2d117a5841dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:30:59.189836 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.189746 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-dshm" (OuterVolumeSpecName: "dshm") pod "f873e236-8ffb-4112-8dcf-2d117a5841dc" (UID: "f873e236-8ffb-4112-8dcf-2d117a5841dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:30:59.189836 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.189766 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f873e236-8ffb-4112-8dcf-2d117a5841dc-kube-api-access-jfghg" (OuterVolumeSpecName: "kube-api-access-jfghg") pod "f873e236-8ffb-4112-8dcf-2d117a5841dc" (UID: "f873e236-8ffb-4112-8dcf-2d117a5841dc"). InnerVolumeSpecName "kube-api-access-jfghg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:30:59.289340 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.289302 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-dshm\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:30:59.289340 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.289336 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jfghg\" (UniqueName: \"kubernetes.io/projected/f873e236-8ffb-4112-8dcf-2d117a5841dc-kube-api-access-jfghg\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:30:59.289340 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.289346 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-tmp-dir\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:30:59.289576 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.289355 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f873e236-8ffb-4112-8dcf-2d117a5841dc-model-cache\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:30:59.289576 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.289364 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f873e236-8ffb-4112-8dcf-2d117a5841dc-tls-certs\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:30:59.951423 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.951337 2564 generic.go:356] "Generic (PLEG): container finished" podID="f873e236-8ffb-4112-8dcf-2d117a5841dc" containerID="be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d" exitCode=0 Jul 30 15:30:59.951423 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.951387 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" Jul 30 15:30:59.951980 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.951425 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" event={"ID":"f873e236-8ffb-4112-8dcf-2d117a5841dc","Type":"ContainerDied","Data":"be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d"} Jul 30 15:30:59.951980 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.951471 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8" event={"ID":"f873e236-8ffb-4112-8dcf-2d117a5841dc","Type":"ContainerDied","Data":"6ad36a0c83b576b909be5a6c58621df7b37199371bb59baedd7f573a008caafb"} Jul 30 15:30:59.951980 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.951489 2564 scope.go:117] "RemoveContainer" containerID="be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d" Jul 30 15:30:59.960636 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.960616 2564 scope.go:117] "RemoveContainer" containerID="be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d" Jul 30 15:30:59.960925 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:30:59.960900 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d\": container with ID starting with be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d not found: ID does not exist" containerID="be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d" Jul 30 15:30:59.960975 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.960934 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d"} err="failed to get container status \"be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d\": rpc error: code = NotFound desc = could not find container \"be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d\": container with ID starting with be13532c3fc1d28f0c36d8390ba705ad89d50260a54070d5d23707b1d0f02f6d not found: ID does not exist" Jul 30 15:30:59.968322 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.968298 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8"] Jul 30 15:30:59.971411 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:30:59.971388 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-2fa717fb/e2e-lifecycle-model-2fa717fb-kserve-7c58584c5f-6jqj8"] Jul 30 15:31:01.491555 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:31:01.491515 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f873e236-8ffb-4112-8dcf-2d117a5841dc" path="/var/lib/kubelet/pods/f873e236-8ffb-4112-8dcf-2d117a5841dc/volumes" Jul 30 15:32:53.820642 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:32:53.820522 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:32:53.834729 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:32:53.834706 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:33:03.103700 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.103663 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6"] Jul 30 15:33:03.104281 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.104265 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f873e236-8ffb-4112-8dcf-2d117a5841dc" containerName="main" Jul 30 15:33:03.104342 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.104285 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f873e236-8ffb-4112-8dcf-2d117a5841dc" containerName="main" Jul 30 15:33:03.104386 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.104374 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f873e236-8ffb-4112-8dcf-2d117a5841dc" containerName="main" Jul 30 15:33:03.107238 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.107219 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.109414 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.109390 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-f03573\"" Jul 30 15:33:03.118680 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.118653 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6"] Jul 30 15:33:03.213520 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.213474 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5020e076-b86d-4b08-8e7e-f581c65df341-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6\" (UID: \"5020e076-b86d-4b08-8e7e-f581c65df341\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.213709 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.213561 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjzkj\" (UniqueName: \"kubernetes.io/projected/5020e076-b86d-4b08-8e7e-f581c65df341-kube-api-access-zjzkj\") pod \"payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6\" (UID: \"5020e076-b86d-4b08-8e7e-f581c65df341\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.314595 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.314552 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5020e076-b86d-4b08-8e7e-f581c65df341-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6\" (UID: \"5020e076-b86d-4b08-8e7e-f581c65df341\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.314780 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.314658 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjzkj\" (UniqueName: \"kubernetes.io/projected/5020e076-b86d-4b08-8e7e-f581c65df341-kube-api-access-zjzkj\") pod \"payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6\" (UID: \"5020e076-b86d-4b08-8e7e-f581c65df341\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.315307 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.315287 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5020e076-b86d-4b08-8e7e-f581c65df341-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6\" (UID: \"5020e076-b86d-4b08-8e7e-f581c65df341\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.322596 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.322569 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjzkj\" (UniqueName: \"kubernetes.io/projected/5020e076-b86d-4b08-8e7e-f581c65df341-kube-api-access-zjzkj\") pod \"payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6\" (UID: \"5020e076-b86d-4b08-8e7e-f581c65df341\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.421916 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.421816 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.570318 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.570284 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6"] Jul 30 15:33:03.572395 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:33:03.572363 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5020e076_b86d_4b08_8e7e_f581c65df341.slice/crio-128559d3ad0e5593856fcd831cb4f22e2c95585b065dd1b3cca89fd3cd93bca0 WatchSource:0}: Error finding container 128559d3ad0e5593856fcd831cb4f22e2c95585b065dd1b3cca89fd3cd93bca0: Status 404 returned error can't find the container with id 128559d3ad0e5593856fcd831cb4f22e2c95585b065dd1b3cca89fd3cd93bca0 Jul 30 15:33:03.724185 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.724068 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" event={"ID":"5020e076-b86d-4b08-8e7e-f581c65df341","Type":"ContainerStarted","Data":"c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251"} Jul 30 15:33:03.724185 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.724142 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" event={"ID":"5020e076-b86d-4b08-8e7e-f581c65df341","Type":"ContainerStarted","Data":"128559d3ad0e5593856fcd831cb4f22e2c95585b065dd1b3cca89fd3cd93bca0"} Jul 30 15:33:03.724384 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.724235 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:33:03.738640 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:03.738590 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" podStartSLOduration=0.738575109 podStartE2EDuration="738.575109ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:33:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:33:03.73804209 +0000 UTC m=+2710.721325448" watchObservedRunningTime="2026-07-30 15:33:03.738575109 +0000 UTC m=+2710.721858490" Jul 30 15:33:14.731910 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:33:14.731874 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:34:13.024066 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.024026 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd"] Jul 30 15:34:13.028128 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.028103 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.030909 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.030887 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f03573\"/\"kube-root-ca.crt\"" Jul 30 15:34:13.031091 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.031052 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f03573\"/\"test-model-f03573-kserve-self-signed-certs\"" Jul 30 15:34:13.031216 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.031138 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-f03573\"/\"openshift-service-ca.crt\"" Jul 30 15:34:13.031216 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.031201 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-f03573\"/\"default-dockercfg-wntnc\"" Jul 30 15:34:13.036937 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.036909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd"] Jul 30 15:34:13.089756 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.089725 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-model-cache\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.089972 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.089779 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxt5j\" (UniqueName: \"kubernetes.io/projected/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-kube-api-access-mxt5j\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.089972 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.089815 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-home\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.089972 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.089852 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tls-certs\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.089972 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.089872 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-dshm\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.089972 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.089937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tmp-dir\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.191447 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.191412 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tmp-dir\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.191663 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.191505 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-model-cache\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.191663 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.191569 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxt5j\" (UniqueName: \"kubernetes.io/projected/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-kube-api-access-mxt5j\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.191663 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.191621 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-home\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.191663 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.191656 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tls-certs\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.191891 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.191687 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-dshm\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.191956 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.191941 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tmp-dir\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.192140 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.192115 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-model-cache\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.192284 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.192247 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-home\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.194404 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.194375 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-dshm\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.194904 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.194881 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tls-certs\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.199683 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.199660 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxt5j\" (UniqueName: \"kubernetes.io/projected/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-kube-api-access-mxt5j\") pod \"test-model-f03573-kserve-f47985497-6nhqd\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.340503 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.340407 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:13.499391 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:13.499365 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd"] Jul 30 15:34:13.501216 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:34:13.501187 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode9a89bf2_a2a4_46a5_99f3_445fa3b12782.slice/crio-83462e3b932e027053caa2c086c421de1c3c8513c2161ae842ee32529587c0a7 WatchSource:0}: Error finding container 83462e3b932e027053caa2c086c421de1c3c8513c2161ae842ee32529587c0a7: Status 404 returned error can't find the container with id 83462e3b932e027053caa2c086c421de1c3c8513c2161ae842ee32529587c0a7 Jul 30 15:34:14.156533 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:14.156499 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" event={"ID":"e9a89bf2-a2a4-46a5-99f3-445fa3b12782","Type":"ContainerStarted","Data":"3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5"} Jul 30 15:34:14.157037 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:14.156555 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" event={"ID":"e9a89bf2-a2a4-46a5-99f3-445fa3b12782","Type":"ContainerStarted","Data":"83462e3b932e027053caa2c086c421de1c3c8513c2161ae842ee32529587c0a7"} Jul 30 15:34:14.176101 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:14.176025 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" podStartSLOduration=1.176006933 podStartE2EDuration="1.176006933s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:34:14.173649306 +0000 UTC m=+2781.156932658" watchObservedRunningTime="2026-07-30 15:34:14.176006933 +0000 UTC m=+2781.159290269" Jul 30 15:34:23.341004 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:23.340910 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:23.341004 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:23.340954 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:23.353510 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:23.353482 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:34:24.231589 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:34:24.231560 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:35:35.950457 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:35.950423 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd"] Jul 30 15:35:41.989022 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:41.988987 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6"] Jul 30 15:35:41.989621 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:41.989333 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" podUID="5020e076-b86d-4b08-8e7e-f581c65df341" containerName="payload-pre-processing" containerID="cri-o://c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251" gracePeriod=30 Jul 30 15:35:42.259466 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.259435 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:35:42.292311 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.292282 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5020e076-b86d-4b08-8e7e-f581c65df341-plugins-config-volume\") pod \"5020e076-b86d-4b08-8e7e-f581c65df341\" (UID: \"5020e076-b86d-4b08-8e7e-f581c65df341\") " Jul 30 15:35:42.292466 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.292322 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zjzkj\" (UniqueName: \"kubernetes.io/projected/5020e076-b86d-4b08-8e7e-f581c65df341-kube-api-access-zjzkj\") pod \"5020e076-b86d-4b08-8e7e-f581c65df341\" (UID: \"5020e076-b86d-4b08-8e7e-f581c65df341\") " Jul 30 15:35:42.292709 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.292678 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5020e076-b86d-4b08-8e7e-f581c65df341-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5020e076-b86d-4b08-8e7e-f581c65df341" (UID: "5020e076-b86d-4b08-8e7e-f581c65df341"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:35:42.294440 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.294416 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5020e076-b86d-4b08-8e7e-f581c65df341-kube-api-access-zjzkj" (OuterVolumeSpecName: "kube-api-access-zjzkj") pod "5020e076-b86d-4b08-8e7e-f581c65df341" (UID: "5020e076-b86d-4b08-8e7e-f581c65df341"). InnerVolumeSpecName "kube-api-access-zjzkj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:35:42.393377 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.393343 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5020e076-b86d-4b08-8e7e-f581c65df341-plugins-config-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:35:42.393377 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.393368 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjzkj\" (UniqueName: \"kubernetes.io/projected/5020e076-b86d-4b08-8e7e-f581c65df341-kube-api-access-zjzkj\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:35:42.709685 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.709596 2564 generic.go:356] "Generic (PLEG): container finished" podID="5020e076-b86d-4b08-8e7e-f581c65df341" containerID="c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251" exitCode=0 Jul 30 15:35:42.709685 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.709639 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" Jul 30 15:35:42.709685 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.709676 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" event={"ID":"5020e076-b86d-4b08-8e7e-f581c65df341","Type":"ContainerDied","Data":"c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251"} Jul 30 15:35:42.709935 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.709732 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6" event={"ID":"5020e076-b86d-4b08-8e7e-f581c65df341","Type":"ContainerDied","Data":"128559d3ad0e5593856fcd831cb4f22e2c95585b065dd1b3cca89fd3cd93bca0"} Jul 30 15:35:42.709935 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.709762 2564 scope.go:117] "RemoveContainer" containerID="c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251" Jul 30 15:35:42.720132 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.720106 2564 scope.go:117] "RemoveContainer" containerID="c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251" Jul 30 15:35:42.720393 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:35:42.720376 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251\": container with ID starting with c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251 not found: ID does not exist" containerID="c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251" Jul 30 15:35:42.720444 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.720404 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251"} err="failed to get container status \"c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251\": rpc error: code = NotFound desc = could not find container \"c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251\": container with ID starting with c9d3cff0849b490bbdcf749a84ef154ae4aefa774dd7add04a9121937c68a251 not found: ID does not exist" Jul 30 15:35:42.731649 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.731623 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6"] Jul 30 15:35:42.735318 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:42.735295 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-f03573-748c77dfd4-crqd6"] Jul 30 15:35:43.491227 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:43.491196 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5020e076-b86d-4b08-8e7e-f581c65df341" path="/var/lib/kubelet/pods/5020e076-b86d-4b08-8e7e-f581c65df341/volumes" Jul 30 15:35:50.971842 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:50.971753 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" podUID="e9a89bf2-a2a4-46a5-99f3-445fa3b12782" containerName="main" containerID="cri-o://3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5" gracePeriod=45 Jul 30 15:35:51.234205 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.234181 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:35:51.279916 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.279882 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-model-cache\") pod \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " Jul 30 15:35:51.279916 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.279920 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tls-certs\") pod \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " Jul 30 15:35:51.280190 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.279975 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-home\") pod \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " Jul 30 15:35:51.280190 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.279996 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mxt5j\" (UniqueName: \"kubernetes.io/projected/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-kube-api-access-mxt5j\") pod \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " Jul 30 15:35:51.280190 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.280021 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tmp-dir\") pod \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " Jul 30 15:35:51.280190 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.280043 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-dshm\") pod \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\" (UID: \"e9a89bf2-a2a4-46a5-99f3-445fa3b12782\") " Jul 30 15:35:51.280402 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.280258 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-model-cache" (OuterVolumeSpecName: "model-cache") pod "e9a89bf2-a2a4-46a5-99f3-445fa3b12782" (UID: "e9a89bf2-a2a4-46a5-99f3-445fa3b12782"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:35:51.280402 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.280288 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-home" (OuterVolumeSpecName: "home") pod "e9a89bf2-a2a4-46a5-99f3-445fa3b12782" (UID: "e9a89bf2-a2a4-46a5-99f3-445fa3b12782"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:35:51.280402 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.280346 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e9a89bf2-a2a4-46a5-99f3-445fa3b12782" (UID: "e9a89bf2-a2a4-46a5-99f3-445fa3b12782"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:35:51.280598 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.280535 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-home\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:35:51.280598 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.280566 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tmp-dir\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:35:51.280598 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.280581 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-model-cache\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:35:51.282380 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.282348 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-kube-api-access-mxt5j" (OuterVolumeSpecName: "kube-api-access-mxt5j") pod "e9a89bf2-a2a4-46a5-99f3-445fa3b12782" (UID: "e9a89bf2-a2a4-46a5-99f3-445fa3b12782"). InnerVolumeSpecName "kube-api-access-mxt5j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:35:51.282504 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.282384 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e9a89bf2-a2a4-46a5-99f3-445fa3b12782" (UID: "e9a89bf2-a2a4-46a5-99f3-445fa3b12782"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:35:51.282504 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.282461 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-dshm" (OuterVolumeSpecName: "dshm") pod "e9a89bf2-a2a4-46a5-99f3-445fa3b12782" (UID: "e9a89bf2-a2a4-46a5-99f3-445fa3b12782"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:35:51.381126 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.381068 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxt5j\" (UniqueName: \"kubernetes.io/projected/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-kube-api-access-mxt5j\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:35:51.381126 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.381123 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-dshm\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:35:51.381332 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.381135 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e9a89bf2-a2a4-46a5-99f3-445fa3b12782-tls-certs\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:35:51.767170 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.767062 2564 generic.go:356] "Generic (PLEG): container finished" podID="e9a89bf2-a2a4-46a5-99f3-445fa3b12782" containerID="3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5" exitCode=0 Jul 30 15:35:51.767402 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.767145 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" Jul 30 15:35:51.767528 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.767169 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" event={"ID":"e9a89bf2-a2a4-46a5-99f3-445fa3b12782","Type":"ContainerDied","Data":"3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5"} Jul 30 15:35:51.767528 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.767517 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd" event={"ID":"e9a89bf2-a2a4-46a5-99f3-445fa3b12782","Type":"ContainerDied","Data":"83462e3b932e027053caa2c086c421de1c3c8513c2161ae842ee32529587c0a7"} Jul 30 15:35:51.767650 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.767545 2564 scope.go:117] "RemoveContainer" containerID="3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5" Jul 30 15:35:51.785143 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.785122 2564 scope.go:117] "RemoveContainer" containerID="3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5" Jul 30 15:35:51.785440 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:35:51.785418 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5\": container with ID starting with 3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5 not found: ID does not exist" containerID="3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5" Jul 30 15:35:51.785490 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.785451 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5"} err="failed to get container status \"3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5\": rpc error: code = NotFound desc = could not find container \"3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5\": container with ID starting with 3ad88b94a98ca722cb313c5b625f2f6984fe15f2323435eb5165cd925052baf5 not found: ID does not exist" Jul 30 15:35:51.792566 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.792542 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd"] Jul 30 15:35:51.795997 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:51.795973 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-f03573/test-model-f03573-kserve-f47985497-6nhqd"] Jul 30 15:35:53.491032 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:35:53.490989 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9a89bf2-a2a4-46a5-99f3-445fa3b12782" path="/var/lib/kubelet/pods/e9a89bf2-a2a4-46a5-99f3-445fa3b12782/volumes" Jul 30 15:36:41.220210 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.220175 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6"] Jul 30 15:36:41.220688 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.220622 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9a89bf2-a2a4-46a5-99f3-445fa3b12782" containerName="main" Jul 30 15:36:41.220688 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.220633 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9a89bf2-a2a4-46a5-99f3-445fa3b12782" containerName="main" Jul 30 15:36:41.220688 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.220650 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5020e076-b86d-4b08-8e7e-f581c65df341" containerName="payload-pre-processing" Jul 30 15:36:41.220688 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.220656 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="5020e076-b86d-4b08-8e7e-f581c65df341" containerName="payload-pre-processing" Jul 30 15:36:41.220825 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.220727 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9a89bf2-a2a4-46a5-99f3-445fa3b12782" containerName="main" Jul 30 15:36:41.220825 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.220739 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="5020e076-b86d-4b08-8e7e-f581c65df341" containerName="payload-pre-processing" Jul 30 15:36:41.223894 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.223873 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:41.226058 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.226036 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-e2257c\"" Jul 30 15:36:41.226179 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.226056 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-e2257c-dockercfg-djv97\"" Jul 30 15:36:41.237761 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.237729 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6"] Jul 30 15:36:41.270321 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.270285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rkmg\" (UniqueName: \"kubernetes.io/projected/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-kube-api-access-7rkmg\") pod \"payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6\" (UID: \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:41.270497 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.270368 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6\" (UID: \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:41.371684 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.371643 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rkmg\" (UniqueName: \"kubernetes.io/projected/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-kube-api-access-7rkmg\") pod \"payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6\" (UID: \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:41.371860 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.371726 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6\" (UID: \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:41.372503 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.372484 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6\" (UID: \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:41.378656 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.378632 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rkmg\" (UniqueName: \"kubernetes.io/projected/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-kube-api-access-7rkmg\") pod \"payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6\" (UID: \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:41.534100 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.534058 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:41.678335 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:41.678303 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6"] Jul 30 15:36:41.679918 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:36:41.679890 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75a56ab5_7a23_4f01_8ab8_c4363e095bb2.slice/crio-60f38189338d4abda8a969a30ab37c042a4a8c2488447884a4d12e7e7de7706f WatchSource:0}: Error finding container 60f38189338d4abda8a969a30ab37c042a4a8c2488447884a4d12e7e7de7706f: Status 404 returned error can't find the container with id 60f38189338d4abda8a969a30ab37c042a4a8c2488447884a4d12e7e7de7706f Jul 30 15:36:42.074507 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:42.074454 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" event={"ID":"75a56ab5-7a23-4f01-8ab8-c4363e095bb2","Type":"ContainerStarted","Data":"3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495"} Jul 30 15:36:42.074721 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:42.074533 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" event={"ID":"75a56ab5-7a23-4f01-8ab8-c4363e095bb2","Type":"ContainerStarted","Data":"60f38189338d4abda8a969a30ab37c042a4a8c2488447884a4d12e7e7de7706f"} Jul 30 15:36:42.074721 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:42.074659 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:36:42.092415 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:42.092364 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" podStartSLOduration=1.092347624 podStartE2EDuration="1.092347624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:36:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:36:42.088950841 +0000 UTC m=+2929.072234206" watchObservedRunningTime="2026-07-30 15:36:42.092347624 +0000 UTC m=+2929.075630997" Jul 30 15:36:53.088269 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:36:53.088236 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:37:46.628361 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:46.628321 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6"] Jul 30 15:37:46.630983 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:46.630883 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" podUID="75a56ab5-7a23-4f01-8ab8-c4363e095bb2" containerName="payload-pre-processing" containerID="cri-o://3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495" gracePeriod=30 Jul 30 15:37:46.911947 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:46.911919 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:37:46.921971 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:46.921941 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-plugins-config-volume\") pod \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\" (UID: \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\") " Jul 30 15:37:46.922154 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:46.921981 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7rkmg\" (UniqueName: \"kubernetes.io/projected/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-kube-api-access-7rkmg\") pod \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\" (UID: \"75a56ab5-7a23-4f01-8ab8-c4363e095bb2\") " Jul 30 15:37:46.922345 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:46.922322 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "75a56ab5-7a23-4f01-8ab8-c4363e095bb2" (UID: "75a56ab5-7a23-4f01-8ab8-c4363e095bb2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:37:46.924050 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:46.924031 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-kube-api-access-7rkmg" (OuterVolumeSpecName: "kube-api-access-7rkmg") pod "75a56ab5-7a23-4f01-8ab8-c4363e095bb2" (UID: "75a56ab5-7a23-4f01-8ab8-c4363e095bb2"). InnerVolumeSpecName "kube-api-access-7rkmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:37:47.023150 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.023116 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-plugins-config-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:37:47.023150 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.023143 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7rkmg\" (UniqueName: \"kubernetes.io/projected/75a56ab5-7a23-4f01-8ab8-c4363e095bb2-kube-api-access-7rkmg\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:37:47.485001 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.484970 2564 generic.go:356] "Generic (PLEG): container finished" podID="75a56ab5-7a23-4f01-8ab8-c4363e095bb2" containerID="3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495" exitCode=0 Jul 30 15:37:47.485204 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.485060 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" Jul 30 15:37:47.490857 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.490831 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" event={"ID":"75a56ab5-7a23-4f01-8ab8-c4363e095bb2","Type":"ContainerDied","Data":"3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495"} Jul 30 15:37:47.490994 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.490875 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6" event={"ID":"75a56ab5-7a23-4f01-8ab8-c4363e095bb2","Type":"ContainerDied","Data":"60f38189338d4abda8a969a30ab37c042a4a8c2488447884a4d12e7e7de7706f"} Jul 30 15:37:47.490994 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.490900 2564 scope.go:117] "RemoveContainer" containerID="3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495" Jul 30 15:37:47.500494 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.500475 2564 scope.go:117] "RemoveContainer" containerID="3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495" Jul 30 15:37:47.500751 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:37:47.500733 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495\": container with ID starting with 3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495 not found: ID does not exist" containerID="3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495" Jul 30 15:37:47.500796 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.500762 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495"} err="failed to get container status \"3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495\": rpc error: code = NotFound desc = could not find container \"3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495\": container with ID starting with 3b8bf22b679c4926e97a6cb8ce35cec4d73047aa776dd38b9cd6bca4e8cb6495 not found: ID does not exist" Jul 30 15:37:47.507866 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.507841 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6"] Jul 30 15:37:47.511334 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:47.511309 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-e2257c-55ffdc98dd-r98c6"] Jul 30 15:37:49.491130 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:49.491066 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75a56ab5-7a23-4f01-8ab8-c4363e095bb2" path="/var/lib/kubelet/pods/75a56ab5-7a23-4f01-8ab8-c4363e095bb2/volumes" Jul 30 15:37:53.860641 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:53.860525 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:37:53.877998 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:37:53.877973 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:38:28.951529 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:28.951490 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h"] Jul 30 15:38:28.952053 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:28.951974 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75a56ab5-7a23-4f01-8ab8-c4363e095bb2" containerName="payload-pre-processing" Jul 30 15:38:28.952053 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:28.951987 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="75a56ab5-7a23-4f01-8ab8-c4363e095bb2" containerName="payload-pre-processing" Jul 30 15:38:28.952189 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:28.952063 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="75a56ab5-7a23-4f01-8ab8-c4363e095bb2" containerName="payload-pre-processing" Jul 30 15:38:28.955066 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:28.955051 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:28.968287 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:28.968259 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-2b1f1f-dockercfg-49xbx\"" Jul 30 15:38:28.968432 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:28.968302 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-2b1f1f\"" Jul 30 15:38:28.976798 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:28.976770 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h"] Jul 30 15:38:29.022524 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.022475 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa351034-712c-445b-8bc0-7483a473997d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h\" (UID: \"fa351034-712c-445b-8bc0-7483a473997d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:29.022524 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.022524 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmhf9\" (UniqueName: \"kubernetes.io/projected/fa351034-712c-445b-8bc0-7483a473997d-kube-api-access-rmhf9\") pod \"payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h\" (UID: \"fa351034-712c-445b-8bc0-7483a473997d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:29.123452 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.123401 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa351034-712c-445b-8bc0-7483a473997d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h\" (UID: \"fa351034-712c-445b-8bc0-7483a473997d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:29.123452 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.123453 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmhf9\" (UniqueName: \"kubernetes.io/projected/fa351034-712c-445b-8bc0-7483a473997d-kube-api-access-rmhf9\") pod \"payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h\" (UID: \"fa351034-712c-445b-8bc0-7483a473997d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:29.124236 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.124216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa351034-712c-445b-8bc0-7483a473997d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h\" (UID: \"fa351034-712c-445b-8bc0-7483a473997d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:29.131361 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.131338 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmhf9\" (UniqueName: \"kubernetes.io/projected/fa351034-712c-445b-8bc0-7483a473997d-kube-api-access-rmhf9\") pod \"payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h\" (UID: \"fa351034-712c-445b-8bc0-7483a473997d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:29.266217 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.266179 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:29.411330 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.411305 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h"] Jul 30 15:38:29.413557 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:38:29.413528 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa351034_712c_445b_8bc0_7483a473997d.slice/crio-574b813f05ceceb0abba334924e39ba6669794e2acecfa9f36049d2f27ff05a8 WatchSource:0}: Error finding container 574b813f05ceceb0abba334924e39ba6669794e2acecfa9f36049d2f27ff05a8: Status 404 returned error can't find the container with id 574b813f05ceceb0abba334924e39ba6669794e2acecfa9f36049d2f27ff05a8 Jul 30 15:38:29.745309 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.745268 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" event={"ID":"fa351034-712c-445b-8bc0-7483a473997d","Type":"ContainerStarted","Data":"7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b"} Jul 30 15:38:29.745530 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.745330 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" event={"ID":"fa351034-712c-445b-8bc0-7483a473997d","Type":"ContainerStarted","Data":"574b813f05ceceb0abba334924e39ba6669794e2acecfa9f36049d2f27ff05a8"} Jul 30 15:38:29.745530 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.745360 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:38:29.762450 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:29.762398 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" podStartSLOduration=1.762383686 podStartE2EDuration="1.762383686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:38:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:38:29.760256638 +0000 UTC m=+3036.743539969" watchObservedRunningTime="2026-07-30 15:38:29.762383686 +0000 UTC m=+3036.745667023" Jul 30 15:38:40.754518 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:38:40.754483 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:39:00.018024 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.017987 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2"] Jul 30 15:39:00.022359 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.022334 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.025028 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.025002 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-845a26\"/\"default-dockercfg-bc8rv\"" Jul 30 15:39:00.025314 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.025001 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-845a26\"/\"openshift-service-ca.crt\"" Jul 30 15:39:00.025429 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.025003 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-845a26\"/\"auth-test-model-845a26-kserve-self-signed-certs\"" Jul 30 15:39:00.025429 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.025053 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-845a26\"/\"kube-root-ca.crt\"" Jul 30 15:39:00.031787 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.031761 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2"] Jul 30 15:39:00.136469 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.136436 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrgrk\" (UniqueName: \"kubernetes.io/projected/0545c285-3fcd-4918-88b9-d30f2b736240-kube-api-access-vrgrk\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.136674 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.136499 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-tmp-dir\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.136674 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.136532 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-home\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.136674 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.136635 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-model-cache\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.136834 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.136703 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0545c285-3fcd-4918-88b9-d30f2b736240-tls-certs\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.136834 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.136759 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-dshm\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.237715 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.237671 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-model-cache\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.237913 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.237734 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0545c285-3fcd-4918-88b9-d30f2b736240-tls-certs\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.237913 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.237777 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-dshm\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.237913 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.237822 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrgrk\" (UniqueName: \"kubernetes.io/projected/0545c285-3fcd-4918-88b9-d30f2b736240-kube-api-access-vrgrk\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.237913 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.237862 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-tmp-dir\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.237913 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.237892 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-home\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.238249 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.238192 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-model-cache\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.238249 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.238234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-tmp-dir\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.238329 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.238275 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-home\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.240178 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.240152 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-dshm\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.240454 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.240433 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0545c285-3fcd-4918-88b9-d30f2b736240-tls-certs\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.244777 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.244750 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrgrk\" (UniqueName: \"kubernetes.io/projected/0545c285-3fcd-4918-88b9-d30f2b736240-kube-api-access-vrgrk\") pod \"auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.332853 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.332764 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:00.478660 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.478637 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2"] Jul 30 15:39:00.480054 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:39:00.480025 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0545c285_3fcd_4918_88b9_d30f2b736240.slice/crio-79a0fb58d810d6c51669c6b0d05dbea25dfdbb4335d1745cd903d80b207bf738 WatchSource:0}: Error finding container 79a0fb58d810d6c51669c6b0d05dbea25dfdbb4335d1745cd903d80b207bf738: Status 404 returned error can't find the container with id 79a0fb58d810d6c51669c6b0d05dbea25dfdbb4335d1745cd903d80b207bf738 Jul 30 15:39:00.939762 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.939726 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" event={"ID":"0545c285-3fcd-4918-88b9-d30f2b736240","Type":"ContainerStarted","Data":"0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e"} Jul 30 15:39:00.939941 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.939774 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" event={"ID":"0545c285-3fcd-4918-88b9-d30f2b736240","Type":"ContainerStarted","Data":"79a0fb58d810d6c51669c6b0d05dbea25dfdbb4335d1745cd903d80b207bf738"} Jul 30 15:39:00.958570 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:00.958515 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" podStartSLOduration=0.958497851 podStartE2EDuration="958.497851ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:39:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:39:00.956680257 +0000 UTC m=+3067.939963596" watchObservedRunningTime="2026-07-30 15:39:00.958497851 +0000 UTC m=+3067.941781204" Jul 30 15:39:10.333651 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:10.333611 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:10.333651 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:10.333663 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:10.346224 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:10.346199 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:11.017575 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:11.017546 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:39:47.263801 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.263762 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66gs8"] Jul 30 15:39:47.267845 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.267814 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:47.276537 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.276500 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66gs8"] Jul 30 15:39:47.315140 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.315067 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bdg9t\" (UniqueName: \"kubernetes.io/projected/093beac3-bc4c-4f8e-9025-d3399d1b06ca-kube-api-access-bdg9t\") pod \"kuadrant-operator-catalog-66gs8\" (UID: \"093beac3-bc4c-4f8e-9025-d3399d1b06ca\") " pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:47.416533 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.416485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bdg9t\" (UniqueName: \"kubernetes.io/projected/093beac3-bc4c-4f8e-9025-d3399d1b06ca-kube-api-access-bdg9t\") pod \"kuadrant-operator-catalog-66gs8\" (UID: \"093beac3-bc4c-4f8e-9025-d3399d1b06ca\") " pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:47.423947 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.423912 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bdg9t\" (UniqueName: \"kubernetes.io/projected/093beac3-bc4c-4f8e-9025-d3399d1b06ca-kube-api-access-bdg9t\") pod \"kuadrant-operator-catalog-66gs8\" (UID: \"093beac3-bc4c-4f8e-9025-d3399d1b06ca\") " pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:47.581726 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.581633 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:47.728121 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.728092 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66gs8"] Jul 30 15:39:47.730006 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:39:47.729976 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod093beac3_bc4c_4f8e_9025_d3399d1b06ca.slice/crio-f6ce897c2db12eaf14febcb33299d13b25cccf562e394bb98f48bc7cba4289fb WatchSource:0}: Error finding container f6ce897c2db12eaf14febcb33299d13b25cccf562e394bb98f48bc7cba4289fb: Status 404 returned error can't find the container with id f6ce897c2db12eaf14febcb33299d13b25cccf562e394bb98f48bc7cba4289fb Jul 30 15:39:47.731385 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:47.731364 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 15:39:48.232640 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:48.232605 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" event={"ID":"093beac3-bc4c-4f8e-9025-d3399d1b06ca","Type":"ContainerStarted","Data":"f6ce897c2db12eaf14febcb33299d13b25cccf562e394bb98f48bc7cba4289fb"} Jul 30 15:39:49.240453 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:49.240417 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" event={"ID":"093beac3-bc4c-4f8e-9025-d3399d1b06ca","Type":"ContainerStarted","Data":"59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113"} Jul 30 15:39:49.255815 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:49.255767 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" podStartSLOduration=1.726174882 podStartE2EDuration="2.255750276s" totalImagesPullingTime="529.575394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:39:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:39:49.254592988 +0000 UTC m=+3116.237876325" watchObservedRunningTime="2026-07-30 15:39:49.255750276 +0000 UTC m=+3116.239033641" Jul 30 15:39:57.582845 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:57.582806 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:57.583395 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:57.582862 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:57.605324 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:57.605297 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:58.317800 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:58.317764 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:39:58.351486 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:39:58.351452 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66gs8"] Jul 30 15:40:00.310842 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:00.310802 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" podUID="093beac3-bc4c-4f8e-9025-d3399d1b06ca" containerName="registry-server" containerID="cri-o://59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113" gracePeriod=2 Jul 30 15:40:00.564620 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:00.564558 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:40:00.655870 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:00.655830 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bdg9t\" (UniqueName: \"kubernetes.io/projected/093beac3-bc4c-4f8e-9025-d3399d1b06ca-kube-api-access-bdg9t\") pod \"093beac3-bc4c-4f8e-9025-d3399d1b06ca\" (UID: \"093beac3-bc4c-4f8e-9025-d3399d1b06ca\") " Jul 30 15:40:00.657971 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:00.657936 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/093beac3-bc4c-4f8e-9025-d3399d1b06ca-kube-api-access-bdg9t" (OuterVolumeSpecName: "kube-api-access-bdg9t") pod "093beac3-bc4c-4f8e-9025-d3399d1b06ca" (UID: "093beac3-bc4c-4f8e-9025-d3399d1b06ca"). InnerVolumeSpecName "kube-api-access-bdg9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:40:00.757479 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:00.757440 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bdg9t\" (UniqueName: \"kubernetes.io/projected/093beac3-bc4c-4f8e-9025-d3399d1b06ca-kube-api-access-bdg9t\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:40:01.317700 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.317669 2564 generic.go:356] "Generic (PLEG): container finished" podID="093beac3-bc4c-4f8e-9025-d3399d1b06ca" containerID="59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113" exitCode=0 Jul 30 15:40:01.318158 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.317713 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" Jul 30 15:40:01.318158 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.317717 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" event={"ID":"093beac3-bc4c-4f8e-9025-d3399d1b06ca","Type":"ContainerDied","Data":"59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113"} Jul 30 15:40:01.318158 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.317833 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-66gs8" event={"ID":"093beac3-bc4c-4f8e-9025-d3399d1b06ca","Type":"ContainerDied","Data":"f6ce897c2db12eaf14febcb33299d13b25cccf562e394bb98f48bc7cba4289fb"} Jul 30 15:40:01.318158 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.317858 2564 scope.go:117] "RemoveContainer" containerID="59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113" Jul 30 15:40:01.328086 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.328055 2564 scope.go:117] "RemoveContainer" containerID="59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113" Jul 30 15:40:01.328463 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:40:01.328433 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113\": container with ID starting with 59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113 not found: ID does not exist" containerID="59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113" Jul 30 15:40:01.328541 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.328477 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113"} err="failed to get container status \"59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113\": rpc error: code = NotFound desc = could not find container \"59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113\": container with ID starting with 59f5717271ccbcd5bb12457fb361a1bcda128a0a82b643bb3d6639758d03e113 not found: ID does not exist" Jul 30 15:40:01.339522 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.339493 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66gs8"] Jul 30 15:40:01.342206 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.342184 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-66gs8"] Jul 30 15:40:01.492820 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:40:01.492781 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="093beac3-bc4c-4f8e-9025-d3399d1b06ca" path="/var/lib/kubelet/pods/093beac3-bc4c-4f8e-9025-d3399d1b06ca/volumes" Jul 30 15:42:53.906456 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:42:53.906346 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:42:53.921633 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:42:53.921604 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:44:18.251500 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:18.251462 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2"] Jul 30 15:44:30.632352 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.632315 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp"] Jul 30 15:44:30.632936 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.632917 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="093beac3-bc4c-4f8e-9025-d3399d1b06ca" containerName="registry-server" Jul 30 15:44:30.633008 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.632940 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="093beac3-bc4c-4f8e-9025-d3399d1b06ca" containerName="registry-server" Jul 30 15:44:30.633048 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.633042 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="093beac3-bc4c-4f8e-9025-d3399d1b06ca" containerName="registry-server" Jul 30 15:44:30.636102 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.636061 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.638201 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.638184 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-845a26\"/\"sub-test-model-845a26-kserve-self-signed-certs\"" Jul 30 15:44:30.644342 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.644316 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp"] Jul 30 15:44:30.805708 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.805667 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f45a72-1295-4d69-9549-117974427d70-tls-certs\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.805708 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.805716 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-dshm\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.805916 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.805797 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-home\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.805916 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.805844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-model-cache\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.805916 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.805876 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-tmp-dir\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.806020 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.805937 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dkq\" (UniqueName: \"kubernetes.io/projected/37f45a72-1295-4d69-9549-117974427d70-kube-api-access-v9dkq\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.906696 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.906612 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-dshm\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.906696 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.906672 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-home\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.906696 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.906697 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-model-cache\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.906965 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.906731 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-tmp-dir\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.906965 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.906772 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dkq\" (UniqueName: \"kubernetes.io/projected/37f45a72-1295-4d69-9549-117974427d70-kube-api-access-v9dkq\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.906965 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.906829 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f45a72-1295-4d69-9549-117974427d70-tls-certs\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.907157 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.907140 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-home\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.907271 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.907200 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-tmp-dir\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.907271 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.907202 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-model-cache\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.909005 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.908977 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-dshm\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.909236 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.909220 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f45a72-1295-4d69-9549-117974427d70-tls-certs\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.915254 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.915234 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dkq\" (UniqueName: \"kubernetes.io/projected/37f45a72-1295-4d69-9549-117974427d70-kube-api-access-v9dkq\") pod \"sub-test-model-845a26-kserve-c46fdcd67-2lkmp\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:30.948365 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:30.948336 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:31.096862 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:31.096654 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp"] Jul 30 15:44:31.099285 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:44:31.099257 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37f45a72_1295_4d69_9549_117974427d70.slice/crio-401f0b9be019ee4a237934f83a546d483aa6be0056ba24781273ee8b3c04e263 WatchSource:0}: Error finding container 401f0b9be019ee4a237934f83a546d483aa6be0056ba24781273ee8b3c04e263: Status 404 returned error can't find the container with id 401f0b9be019ee4a237934f83a546d483aa6be0056ba24781273ee8b3c04e263 Jul 30 15:44:31.995959 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:31.995924 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" event={"ID":"37f45a72-1295-4d69-9549-117974427d70","Type":"ContainerStarted","Data":"b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92"} Jul 30 15:44:31.996385 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:31.995973 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" event={"ID":"37f45a72-1295-4d69-9549-117974427d70","Type":"ContainerStarted","Data":"401f0b9be019ee4a237934f83a546d483aa6be0056ba24781273ee8b3c04e263"} Jul 30 15:44:32.012245 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:32.012190 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" podStartSLOduration=2.012170904 podStartE2EDuration="2.012170904s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:44:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:44:32.011739366 +0000 UTC m=+3398.995022716" watchObservedRunningTime="2026-07-30 15:44:32.012170904 +0000 UTC m=+3398.995454245" Jul 30 15:44:33.274341 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.274300 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" podUID="0545c285-3fcd-4918-88b9-d30f2b736240" containerName="main" containerID="cri-o://0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e" gracePeriod=45 Jul 30 15:44:33.547443 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.547412 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:44:33.634734 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.634700 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-model-cache\") pod \"0545c285-3fcd-4918-88b9-d30f2b736240\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " Jul 30 15:44:33.634734 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.634734 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0545c285-3fcd-4918-88b9-d30f2b736240-tls-certs\") pod \"0545c285-3fcd-4918-88b9-d30f2b736240\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " Jul 30 15:44:33.634970 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.634764 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-dshm\") pod \"0545c285-3fcd-4918-88b9-d30f2b736240\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " Jul 30 15:44:33.634970 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.634786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-tmp-dir\") pod \"0545c285-3fcd-4918-88b9-d30f2b736240\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " Jul 30 15:44:33.634970 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.634832 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-home\") pod \"0545c285-3fcd-4918-88b9-d30f2b736240\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " Jul 30 15:44:33.634970 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.634872 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vrgrk\" (UniqueName: \"kubernetes.io/projected/0545c285-3fcd-4918-88b9-d30f2b736240-kube-api-access-vrgrk\") pod \"0545c285-3fcd-4918-88b9-d30f2b736240\" (UID: \"0545c285-3fcd-4918-88b9-d30f2b736240\") " Jul 30 15:44:33.635197 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.635050 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-model-cache" (OuterVolumeSpecName: "model-cache") pod "0545c285-3fcd-4918-88b9-d30f2b736240" (UID: "0545c285-3fcd-4918-88b9-d30f2b736240"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:33.635197 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.635100 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0545c285-3fcd-4918-88b9-d30f2b736240" (UID: "0545c285-3fcd-4918-88b9-d30f2b736240"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:33.635197 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.635123 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-home" (OuterVolumeSpecName: "home") pod "0545c285-3fcd-4918-88b9-d30f2b736240" (UID: "0545c285-3fcd-4918-88b9-d30f2b736240"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:33.635355 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.635221 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-model-cache\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:44:33.635355 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.635231 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-tmp-dir\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:44:33.635355 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.635240 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-home\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:44:33.637065 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.637044 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-dshm" (OuterVolumeSpecName: "dshm") pod "0545c285-3fcd-4918-88b9-d30f2b736240" (UID: "0545c285-3fcd-4918-88b9-d30f2b736240"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:44:33.637167 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.637066 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0545c285-3fcd-4918-88b9-d30f2b736240-kube-api-access-vrgrk" (OuterVolumeSpecName: "kube-api-access-vrgrk") pod "0545c285-3fcd-4918-88b9-d30f2b736240" (UID: "0545c285-3fcd-4918-88b9-d30f2b736240"). InnerVolumeSpecName "kube-api-access-vrgrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:44:33.637167 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.637128 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0545c285-3fcd-4918-88b9-d30f2b736240-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0545c285-3fcd-4918-88b9-d30f2b736240" (UID: "0545c285-3fcd-4918-88b9-d30f2b736240"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:44:33.735903 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.735850 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0545c285-3fcd-4918-88b9-d30f2b736240-tls-certs\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:44:33.735903 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.735897 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0545c285-3fcd-4918-88b9-d30f2b736240-dshm\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:44:33.735903 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:33.735909 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrgrk\" (UniqueName: \"kubernetes.io/projected/0545c285-3fcd-4918-88b9-d30f2b736240-kube-api-access-vrgrk\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:44:34.009523 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.009487 2564 generic.go:356] "Generic (PLEG): container finished" podID="0545c285-3fcd-4918-88b9-d30f2b736240" containerID="0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e" exitCode=0 Jul 30 15:44:34.009708 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.009547 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" Jul 30 15:44:34.009708 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.009574 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" event={"ID":"0545c285-3fcd-4918-88b9-d30f2b736240","Type":"ContainerDied","Data":"0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e"} Jul 30 15:44:34.009708 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.009621 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2" event={"ID":"0545c285-3fcd-4918-88b9-d30f2b736240","Type":"ContainerDied","Data":"79a0fb58d810d6c51669c6b0d05dbea25dfdbb4335d1745cd903d80b207bf738"} Jul 30 15:44:34.009708 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.009638 2564 scope.go:117] "RemoveContainer" containerID="0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e" Jul 30 15:44:34.019305 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.019287 2564 scope.go:117] "RemoveContainer" containerID="0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e" Jul 30 15:44:34.019557 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:44:34.019538 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e\": container with ID starting with 0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e not found: ID does not exist" containerID="0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e" Jul 30 15:44:34.019611 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.019573 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e"} err="failed to get container status \"0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e\": rpc error: code = NotFound desc = could not find container \"0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e\": container with ID starting with 0b1ed3b5bcc8f1a68aecf3354a2c1fd77d25e19e4a532604cbc9f89762b1b38e not found: ID does not exist" Jul 30 15:44:34.030830 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.030804 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2"] Jul 30 15:44:34.034720 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:34.034696 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/auth-test-model-845a26-kserve-7dd5c87f5f-bjpr2"] Jul 30 15:44:35.491819 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:35.491782 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0545c285-3fcd-4918-88b9-d30f2b736240" path="/var/lib/kubelet/pods/0545c285-3fcd-4918-88b9-d30f2b736240/volumes" Jul 30 15:44:40.948731 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:40.948687 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:40.949300 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:40.948756 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:40.961554 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:40.961527 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:44:41.066784 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:44:41.066755 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:46:51.187311 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:46:51.187277 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp"] Jul 30 15:47:06.206833 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.206762 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" podUID="37f45a72-1295-4d69-9549-117974427d70" containerName="main" containerID="cri-o://b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92" gracePeriod=45 Jul 30 15:47:06.481523 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.481497 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:47:06.586334 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586301 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-home\") pod \"37f45a72-1295-4d69-9549-117974427d70\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " Jul 30 15:47:06.586515 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586446 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-model-cache\") pod \"37f45a72-1295-4d69-9549-117974427d70\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " Jul 30 15:47:06.586515 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586470 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v9dkq\" (UniqueName: \"kubernetes.io/projected/37f45a72-1295-4d69-9549-117974427d70-kube-api-access-v9dkq\") pod \"37f45a72-1295-4d69-9549-117974427d70\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " Jul 30 15:47:06.586515 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586502 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-dshm\") pod \"37f45a72-1295-4d69-9549-117974427d70\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " Jul 30 15:47:06.586689 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586528 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f45a72-1295-4d69-9549-117974427d70-tls-certs\") pod \"37f45a72-1295-4d69-9549-117974427d70\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " Jul 30 15:47:06.586689 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586552 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-tmp-dir\") pod \"37f45a72-1295-4d69-9549-117974427d70\" (UID: \"37f45a72-1295-4d69-9549-117974427d70\") " Jul 30 15:47:06.586689 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586551 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-home" (OuterVolumeSpecName: "home") pod "37f45a72-1295-4d69-9549-117974427d70" (UID: "37f45a72-1295-4d69-9549-117974427d70"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:47:06.586853 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586757 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-model-cache" (OuterVolumeSpecName: "model-cache") pod "37f45a72-1295-4d69-9549-117974427d70" (UID: "37f45a72-1295-4d69-9549-117974427d70"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:47:06.586853 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586830 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "37f45a72-1295-4d69-9549-117974427d70" (UID: "37f45a72-1295-4d69-9549-117974427d70"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:47:06.586947 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586886 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-model-cache\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:47:06.586947 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.586902 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-home\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:47:06.588606 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.588576 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37f45a72-1295-4d69-9549-117974427d70-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "37f45a72-1295-4d69-9549-117974427d70" (UID: "37f45a72-1295-4d69-9549-117974427d70"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:47:06.589028 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.589002 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37f45a72-1295-4d69-9549-117974427d70-kube-api-access-v9dkq" (OuterVolumeSpecName: "kube-api-access-v9dkq") pod "37f45a72-1295-4d69-9549-117974427d70" (UID: "37f45a72-1295-4d69-9549-117974427d70"). InnerVolumeSpecName "kube-api-access-v9dkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:47:06.589207 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.589062 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-dshm" (OuterVolumeSpecName: "dshm") pod "37f45a72-1295-4d69-9549-117974427d70" (UID: "37f45a72-1295-4d69-9549-117974427d70"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:47:06.688210 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.688174 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9dkq\" (UniqueName: \"kubernetes.io/projected/37f45a72-1295-4d69-9549-117974427d70-kube-api-access-v9dkq\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:47:06.688210 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.688207 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-dshm\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:47:06.688210 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.688219 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37f45a72-1295-4d69-9549-117974427d70-tls-certs\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:47:06.688571 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.688248 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37f45a72-1295-4d69-9549-117974427d70-tmp-dir\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:47:06.945333 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.945298 2564 generic.go:356] "Generic (PLEG): container finished" podID="37f45a72-1295-4d69-9549-117974427d70" containerID="b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92" exitCode=0 Jul 30 15:47:06.945509 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.945343 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" Jul 30 15:47:06.945509 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.945380 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" event={"ID":"37f45a72-1295-4d69-9549-117974427d70","Type":"ContainerDied","Data":"b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92"} Jul 30 15:47:06.945509 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.945422 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp" event={"ID":"37f45a72-1295-4d69-9549-117974427d70","Type":"ContainerDied","Data":"401f0b9be019ee4a237934f83a546d483aa6be0056ba24781273ee8b3c04e263"} Jul 30 15:47:06.945509 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.945440 2564 scope.go:117] "RemoveContainer" containerID="b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92" Jul 30 15:47:06.957660 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.957637 2564 scope.go:117] "RemoveContainer" containerID="b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92" Jul 30 15:47:06.957960 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:47:06.957936 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92\": container with ID starting with b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92 not found: ID does not exist" containerID="b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92" Jul 30 15:47:06.958068 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.957966 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92"} err="failed to get container status \"b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92\": rpc error: code = NotFound desc = could not find container \"b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92\": container with ID starting with b381566eca4e8c32de04bc96f174a9d0e5f7ecd6192ad9aba5b3ce76b604eb92 not found: ID does not exist" Jul 30 15:47:06.969225 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.969198 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp"] Jul 30 15:47:06.972685 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:06.972659 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-845a26/sub-test-model-845a26-kserve-c46fdcd67-2lkmp"] Jul 30 15:47:07.491853 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:07.491816 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37f45a72-1295-4d69-9549-117974427d70" path="/var/lib/kubelet/pods/37f45a72-1295-4d69-9549-117974427d70/volumes" Jul 30 15:47:40.832446 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.832406 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2"] Jul 30 15:47:40.833090 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.832892 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37f45a72-1295-4d69-9549-117974427d70" containerName="main" Jul 30 15:47:40.833090 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.832906 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="37f45a72-1295-4d69-9549-117974427d70" containerName="main" Jul 30 15:47:40.833090 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.832917 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0545c285-3fcd-4918-88b9-d30f2b736240" containerName="main" Jul 30 15:47:40.833090 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.832926 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="0545c285-3fcd-4918-88b9-d30f2b736240" containerName="main" Jul 30 15:47:40.833090 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.833018 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="0545c285-3fcd-4918-88b9-d30f2b736240" containerName="main" Jul 30 15:47:40.833090 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.833032 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="37f45a72-1295-4d69-9549-117974427d70" containerName="main" Jul 30 15:47:40.836087 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.836054 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:40.838061 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.838037 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2b1f1f\"/\"openshift-service-ca.crt\"" Jul 30 15:47:40.838210 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.838067 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-2b1f1f\"/\"kube-root-ca.crt\"" Jul 30 15:47:40.839889 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.839865 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2b1f1f\"/\"default-dockercfg-fc948\"" Jul 30 15:47:40.839889 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.839885 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-2b1f1f\"/\"rate-test-model-2b1f1f-kserve-self-signed-certs\"" Jul 30 15:47:40.846166 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.846142 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2"] Jul 30 15:47:40.920756 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.920715 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-model-cache\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:40.920929 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.920777 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wqwp\" (UniqueName: \"kubernetes.io/projected/1df3097c-87ee-4819-bf01-810b4414194c-kube-api-access-6wqwp\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:40.920929 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.920805 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-home\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:40.920929 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.920844 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-dshm\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:40.920929 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.920862 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df3097c-87ee-4819-bf01-810b4414194c-tls-certs\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:40.920929 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:40.920879 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-tmp-dir\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022180 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022143 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-home\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022380 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022192 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-dshm\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022380 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022322 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df3097c-87ee-4819-bf01-810b4414194c-tls-certs\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022380 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022370 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-tmp-dir\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022562 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022485 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-model-cache\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022616 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-home\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022616 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022597 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wqwp\" (UniqueName: \"kubernetes.io/projected/1df3097c-87ee-4819-bf01-810b4414194c-kube-api-access-6wqwp\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022719 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022630 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-tmp-dir\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.022817 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.022794 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-model-cache\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.024607 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.024584 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-dshm\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.024698 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.024682 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df3097c-87ee-4819-bf01-810b4414194c-tls-certs\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.030910 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.030892 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wqwp\" (UniqueName: \"kubernetes.io/projected/1df3097c-87ee-4819-bf01-810b4414194c-kube-api-access-6wqwp\") pod \"rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.148323 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.148236 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:41.294831 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:41.294796 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2"] Jul 30 15:47:41.296299 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:47:41.296273 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1df3097c_87ee_4819_bf01_810b4414194c.slice/crio-44ded4c1652fddd65dbd7725ca46c48ddfe68b743fe5292d56a51fc6e8291c1e WatchSource:0}: Error finding container 44ded4c1652fddd65dbd7725ca46c48ddfe68b743fe5292d56a51fc6e8291c1e: Status 404 returned error can't find the container with id 44ded4c1652fddd65dbd7725ca46c48ddfe68b743fe5292d56a51fc6e8291c1e Jul 30 15:47:42.158841 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:42.158787 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" event={"ID":"1df3097c-87ee-4819-bf01-810b4414194c","Type":"ContainerStarted","Data":"e90fe9e99e026f617985aff77bc9857bf4a635796fc81ca3179dae1c7b9d3e4c"} Jul 30 15:47:42.158841 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:42.158841 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" event={"ID":"1df3097c-87ee-4819-bf01-810b4414194c","Type":"ContainerStarted","Data":"44ded4c1652fddd65dbd7725ca46c48ddfe68b743fe5292d56a51fc6e8291c1e"} Jul 30 15:47:42.175291 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:42.175231 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" podStartSLOduration=2.175210046 podStartE2EDuration="2.175210046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:47:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:47:42.174285155 +0000 UTC m=+3589.157568491" watchObservedRunningTime="2026-07-30 15:47:42.175210046 +0000 UTC m=+3589.158493386" Jul 30 15:47:51.148763 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:51.148712 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:51.149259 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:51.148780 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:51.161651 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:51.161624 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:51.226332 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:51.226304 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:47:53.950007 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:53.949894 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:47:53.978209 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:47:53.976574 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:49:33.808200 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:33.808122 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2"] Jul 30 15:49:34.885847 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:34.885806 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7548-139703280696752"] Jul 30 15:49:34.889590 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:34.889569 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280696752" Jul 30 15:49:34.892216 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:34.892195 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kpckp\"" Jul 30 15:49:34.896268 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:34.896197 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7548-139703280696752"] Jul 30 15:49:35.061426 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:35.061388 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shhp9\" (UniqueName: \"kubernetes.io/projected/199c950d-c240-4bbf-aee3-8c2c91173af2-kube-api-access-shhp9\") pod \"test-curl-7548-139703280696752\" (UID: \"199c950d-c240-4bbf-aee3-8c2c91173af2\") " pod="opendatahub/test-curl-7548-139703280696752" Jul 30 15:49:35.162580 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:35.162480 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shhp9\" (UniqueName: \"kubernetes.io/projected/199c950d-c240-4bbf-aee3-8c2c91173af2-kube-api-access-shhp9\") pod \"test-curl-7548-139703280696752\" (UID: \"199c950d-c240-4bbf-aee3-8c2c91173af2\") " pod="opendatahub/test-curl-7548-139703280696752" Jul 30 15:49:35.170450 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:35.170416 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shhp9\" (UniqueName: \"kubernetes.io/projected/199c950d-c240-4bbf-aee3-8c2c91173af2-kube-api-access-shhp9\") pod \"test-curl-7548-139703280696752\" (UID: \"199c950d-c240-4bbf-aee3-8c2c91173af2\") " pod="opendatahub/test-curl-7548-139703280696752" Jul 30 15:49:35.200165 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:35.200136 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280696752" Jul 30 15:49:35.342890 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:35.342865 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7548-139703280696752"] Jul 30 15:49:35.345018 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:49:35.344987 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod199c950d_c240_4bbf_aee3_8c2c91173af2.slice/crio-513a913d90234c89a4910144a636ee30ecdda8a2ce4c87715f9f4d5dad4895ca WatchSource:0}: Error finding container 513a913d90234c89a4910144a636ee30ecdda8a2ce4c87715f9f4d5dad4895ca: Status 404 returned error can't find the container with id 513a913d90234c89a4910144a636ee30ecdda8a2ce4c87715f9f4d5dad4895ca Jul 30 15:49:35.346374 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:35.346357 2564 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 30 15:49:35.863607 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:35.863564 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7548-139703280696752" event={"ID":"199c950d-c240-4bbf-aee3-8c2c91173af2","Type":"ContainerStarted","Data":"513a913d90234c89a4910144a636ee30ecdda8a2ce4c87715f9f4d5dad4895ca"} Jul 30 15:49:36.871460 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:36.871426 2564 generic.go:356] "Generic (PLEG): container finished" podID="199c950d-c240-4bbf-aee3-8c2c91173af2" containerID="cf156b146efdf5a466a3c109cd0617d14b1445d2e6e7a38509275665ebe07c0d" exitCode=0 Jul 30 15:49:36.871872 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:36.871518 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7548-139703280696752" event={"ID":"199c950d-c240-4bbf-aee3-8c2c91173af2","Type":"ContainerDied","Data":"cf156b146efdf5a466a3c109cd0617d14b1445d2e6e7a38509275665ebe07c0d"} Jul 30 15:49:38.018977 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.018947 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280696752" Jul 30 15:49:38.062105 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.062058 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7548-139703280696752_199c950d-c240-4bbf-aee3-8c2c91173af2/test-curl-7548-139703280696752/0.log" Jul 30 15:49:38.088613 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.088579 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7548-139703280696752"] Jul 30 15:49:38.091941 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.091917 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shhp9\" (UniqueName: \"kubernetes.io/projected/199c950d-c240-4bbf-aee3-8c2c91173af2-kube-api-access-shhp9\") pod \"199c950d-c240-4bbf-aee3-8c2c91173af2\" (UID: \"199c950d-c240-4bbf-aee3-8c2c91173af2\") " Jul 30 15:49:38.092697 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.092676 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7548-139703280696752"] Jul 30 15:49:38.094109 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.094085 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/199c950d-c240-4bbf-aee3-8c2c91173af2-kube-api-access-shhp9" (OuterVolumeSpecName: "kube-api-access-shhp9") pod "199c950d-c240-4bbf-aee3-8c2c91173af2" (UID: "199c950d-c240-4bbf-aee3-8c2c91173af2"). InnerVolumeSpecName "kube-api-access-shhp9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:49:38.193631 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.193532 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shhp9\" (UniqueName: \"kubernetes.io/projected/199c950d-c240-4bbf-aee3-8c2c91173af2-kube-api-access-shhp9\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:38.885156 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.885119 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280696752" Jul 30 15:49:38.885344 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:38.885128 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="513a913d90234c89a4910144a636ee30ecdda8a2ce4c87715f9f4d5dad4895ca" Jul 30 15:49:38.894665 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:49:38.894629 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7548-139703280696752\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="199c950d-c240-4bbf-aee3-8c2c91173af2" pod="opendatahub/test-curl-7548-139703280696752" Jul 30 15:49:39.493664 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:39.493628 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="199c950d-c240-4bbf-aee3-8c2c91173af2" path="/var/lib/kubelet/pods/199c950d-c240-4bbf-aee3-8c2c91173af2/volumes" Jul 30 15:49:40.818761 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:40.818723 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7548-139703280203936"] Jul 30 15:49:40.819223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:40.819209 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="199c950d-c240-4bbf-aee3-8c2c91173af2" containerName="test-curl-7548-139703280696752" Jul 30 15:49:40.819223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:40.819224 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="199c950d-c240-4bbf-aee3-8c2c91173af2" containerName="test-curl-7548-139703280696752" Jul 30 15:49:40.819327 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:40.819316 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="199c950d-c240-4bbf-aee3-8c2c91173af2" containerName="test-curl-7548-139703280696752" Jul 30 15:49:40.822765 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:40.822745 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280203936" Jul 30 15:49:40.824920 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:40.824899 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kpckp\"" Jul 30 15:49:40.826933 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:40.826909 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7548-139703280203936"] Jul 30 15:49:40.920784 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:40.920752 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmlz\" (UniqueName: \"kubernetes.io/projected/96766806-87e2-4db5-be94-f887eb679ba3-kube-api-access-jhmlz\") pod \"test-curl-7548-139703280203936\" (UID: \"96766806-87e2-4db5-be94-f887eb679ba3\") " pod="opendatahub/test-curl-7548-139703280203936" Jul 30 15:49:41.022136 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:41.022064 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmlz\" (UniqueName: \"kubernetes.io/projected/96766806-87e2-4db5-be94-f887eb679ba3-kube-api-access-jhmlz\") pod \"test-curl-7548-139703280203936\" (UID: \"96766806-87e2-4db5-be94-f887eb679ba3\") " pod="opendatahub/test-curl-7548-139703280203936" Jul 30 15:49:41.030252 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:41.030219 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmlz\" (UniqueName: \"kubernetes.io/projected/96766806-87e2-4db5-be94-f887eb679ba3-kube-api-access-jhmlz\") pod \"test-curl-7548-139703280203936\" (UID: \"96766806-87e2-4db5-be94-f887eb679ba3\") " pod="opendatahub/test-curl-7548-139703280203936" Jul 30 15:49:41.134142 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:41.134016 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280203936" Jul 30 15:49:41.276496 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:41.276469 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7548-139703280203936"] Jul 30 15:49:41.278596 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:49:41.278558 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod96766806_87e2_4db5_be94_f887eb679ba3.slice/crio-c926196c5fae58a2d34fc7c699a64932c1bd7f161b620a0f5ee0180c77abfb94 WatchSource:0}: Error finding container c926196c5fae58a2d34fc7c699a64932c1bd7f161b620a0f5ee0180c77abfb94: Status 404 returned error can't find the container with id c926196c5fae58a2d34fc7c699a64932c1bd7f161b620a0f5ee0180c77abfb94 Jul 30 15:49:41.904720 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:41.904631 2564 generic.go:356] "Generic (PLEG): container finished" podID="96766806-87e2-4db5-be94-f887eb679ba3" containerID="4cef14c4b5daf5defc2a5647d7444016e0fc9205b244928ad7eb6ba75a992621" exitCode=0 Jul 30 15:49:41.905136 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:41.904722 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7548-139703280203936" event={"ID":"96766806-87e2-4db5-be94-f887eb679ba3","Type":"ContainerDied","Data":"4cef14c4b5daf5defc2a5647d7444016e0fc9205b244928ad7eb6ba75a992621"} Jul 30 15:49:41.905136 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:41.904765 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7548-139703280203936" event={"ID":"96766806-87e2-4db5-be94-f887eb679ba3","Type":"ContainerStarted","Data":"c926196c5fae58a2d34fc7c699a64932c1bd7f161b620a0f5ee0180c77abfb94"} Jul 30 15:49:43.062012 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.061982 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280203936" Jul 30 15:49:43.083193 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.083161 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7548-139703280203936_96766806-87e2-4db5-be94-f887eb679ba3/test-curl-7548-139703280203936/0.log" Jul 30 15:49:43.107761 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.107731 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7548-139703280203936"] Jul 30 15:49:43.110180 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.110157 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7548-139703280203936"] Jul 30 15:49:43.143976 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.143883 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jhmlz\" (UniqueName: \"kubernetes.io/projected/96766806-87e2-4db5-be94-f887eb679ba3-kube-api-access-jhmlz\") pod \"96766806-87e2-4db5-be94-f887eb679ba3\" (UID: \"96766806-87e2-4db5-be94-f887eb679ba3\") " Jul 30 15:49:43.146056 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.146025 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96766806-87e2-4db5-be94-f887eb679ba3-kube-api-access-jhmlz" (OuterVolumeSpecName: "kube-api-access-jhmlz") pod "96766806-87e2-4db5-be94-f887eb679ba3" (UID: "96766806-87e2-4db5-be94-f887eb679ba3"). InnerVolumeSpecName "kube-api-access-jhmlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:49:43.245641 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.245601 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmlz\" (UniqueName: \"kubernetes.io/projected/96766806-87e2-4db5-be94-f887eb679ba3-kube-api-access-jhmlz\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:43.491926 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.491890 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96766806-87e2-4db5-be94-f887eb679ba3" path="/var/lib/kubelet/pods/96766806-87e2-4db5-be94-f887eb679ba3/volumes" Jul 30 15:49:43.918223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.918143 2564 scope.go:117] "RemoveContainer" containerID="4cef14c4b5daf5defc2a5647d7444016e0fc9205b244928ad7eb6ba75a992621" Jul 30 15:49:43.918223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:43.918179 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280203936" Jul 30 15:49:45.810338 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.810299 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7548-139703280859152"] Jul 30 15:49:45.810802 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.810788 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96766806-87e2-4db5-be94-f887eb679ba3" containerName="test-curl-7548-139703280203936" Jul 30 15:49:45.810849 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.810804 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="96766806-87e2-4db5-be94-f887eb679ba3" containerName="test-curl-7548-139703280203936" Jul 30 15:49:45.810891 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.810881 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="96766806-87e2-4db5-be94-f887eb679ba3" containerName="test-curl-7548-139703280203936" Jul 30 15:49:45.815372 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.815352 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:45.817841 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.817818 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-kpckp\"" Jul 30 15:49:45.820885 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.820867 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7548-139703280859152"] Jul 30 15:49:45.874483 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.874446 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94bv\" (UniqueName: \"kubernetes.io/projected/559a3ab7-c752-48ef-bcdd-9ab454d37f84-kube-api-access-t94bv\") pod \"test-curl-7548-139703280859152\" (UID: \"559a3ab7-c752-48ef-bcdd-9ab454d37f84\") " pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:45.975359 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.975323 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94bv\" (UniqueName: \"kubernetes.io/projected/559a3ab7-c752-48ef-bcdd-9ab454d37f84-kube-api-access-t94bv\") pod \"test-curl-7548-139703280859152\" (UID: \"559a3ab7-c752-48ef-bcdd-9ab454d37f84\") " pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:45.983187 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:45.983162 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94bv\" (UniqueName: \"kubernetes.io/projected/559a3ab7-c752-48ef-bcdd-9ab454d37f84-kube-api-access-t94bv\") pod \"test-curl-7548-139703280859152\" (UID: \"559a3ab7-c752-48ef-bcdd-9ab454d37f84\") " pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:46.126874 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:46.126789 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:46.270468 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:46.270444 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7548-139703280859152"] Jul 30 15:49:46.272597 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:49:46.272571 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod559a3ab7_c752_48ef_bcdd_9ab454d37f84.slice/crio-88ee4ec37c3374aa3ffa4dbe529d65c1b2fd07ef15b9531460987bf8ea1cf5a1 WatchSource:0}: Error finding container 88ee4ec37c3374aa3ffa4dbe529d65c1b2fd07ef15b9531460987bf8ea1cf5a1: Status 404 returned error can't find the container with id 88ee4ec37c3374aa3ffa4dbe529d65c1b2fd07ef15b9531460987bf8ea1cf5a1 Jul 30 15:49:46.939359 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:46.939265 2564 generic.go:356] "Generic (PLEG): container finished" podID="559a3ab7-c752-48ef-bcdd-9ab454d37f84" containerID="c928a5cb46f3bf1fd1556cb95f822bfce12b8391aecc65b6dc5bd97dee5d17f1" exitCode=0 Jul 30 15:49:46.939738 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:46.939358 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7548-139703280859152" event={"ID":"559a3ab7-c752-48ef-bcdd-9ab454d37f84","Type":"ContainerDied","Data":"c928a5cb46f3bf1fd1556cb95f822bfce12b8391aecc65b6dc5bd97dee5d17f1"} Jul 30 15:49:46.939738 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:46.939402 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7548-139703280859152" event={"ID":"559a3ab7-c752-48ef-bcdd-9ab454d37f84","Type":"ContainerStarted","Data":"88ee4ec37c3374aa3ffa4dbe529d65c1b2fd07ef15b9531460987bf8ea1cf5a1"} Jul 30 15:49:48.092003 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.091979 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:48.112830 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.112800 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7548-139703280859152_559a3ab7-c752-48ef-bcdd-9ab454d37f84/test-curl-7548-139703280859152/0.log" Jul 30 15:49:48.140592 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.140550 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7548-139703280859152"] Jul 30 15:49:48.146318 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.146294 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7548-139703280859152"] Jul 30 15:49:48.199600 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.199518 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t94bv\" (UniqueName: \"kubernetes.io/projected/559a3ab7-c752-48ef-bcdd-9ab454d37f84-kube-api-access-t94bv\") pod \"559a3ab7-c752-48ef-bcdd-9ab454d37f84\" (UID: \"559a3ab7-c752-48ef-bcdd-9ab454d37f84\") " Jul 30 15:49:48.201559 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.201530 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/559a3ab7-c752-48ef-bcdd-9ab454d37f84-kube-api-access-t94bv" (OuterVolumeSpecName: "kube-api-access-t94bv") pod "559a3ab7-c752-48ef-bcdd-9ab454d37f84" (UID: "559a3ab7-c752-48ef-bcdd-9ab454d37f84"). InnerVolumeSpecName "kube-api-access-t94bv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:49:48.300884 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.300854 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t94bv\" (UniqueName: \"kubernetes.io/projected/559a3ab7-c752-48ef-bcdd-9ab454d37f84-kube-api-access-t94bv\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:48.828182 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.828138 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" podUID="1df3097c-87ee-4819-bf01-810b4414194c" containerName="main" containerID="cri-o://e90fe9e99e026f617985aff77bc9857bf4a635796fc81ca3179dae1c7b9d3e4c" gracePeriod=45 Jul 30 15:49:48.953852 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.953821 2564 generic.go:356] "Generic (PLEG): container finished" podID="1df3097c-87ee-4819-bf01-810b4414194c" containerID="e90fe9e99e026f617985aff77bc9857bf4a635796fc81ca3179dae1c7b9d3e4c" exitCode=0 Jul 30 15:49:48.953998 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.953896 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" event={"ID":"1df3097c-87ee-4819-bf01-810b4414194c","Type":"ContainerDied","Data":"e90fe9e99e026f617985aff77bc9857bf4a635796fc81ca3179dae1c7b9d3e4c"} Jul 30 15:49:48.955268 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.955250 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:48.955383 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:48.955292 2564 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88ee4ec37c3374aa3ffa4dbe529d65c1b2fd07ef15b9531460987bf8ea1cf5a1" Jul 30 15:49:48.967675 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:49:48.967638 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7548-139703280859152\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="559a3ab7-c752-48ef-bcdd-9ab454d37f84" pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:49.091517 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.091456 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:49:49.111142 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:49:49.111102 2564 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7548-139703280859152\" is forbidden: User \"system:node:ip-10-0-130-38.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-38.ec2.internal' and this object" podUID="559a3ab7-c752-48ef-bcdd-9ab454d37f84" pod="opendatahub/test-curl-7548-139703280859152" Jul 30 15:49:49.209606 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.209574 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df3097c-87ee-4819-bf01-810b4414194c-tls-certs\") pod \"1df3097c-87ee-4819-bf01-810b4414194c\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " Jul 30 15:49:49.209788 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.209638 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wqwp\" (UniqueName: \"kubernetes.io/projected/1df3097c-87ee-4819-bf01-810b4414194c-kube-api-access-6wqwp\") pod \"1df3097c-87ee-4819-bf01-810b4414194c\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " Jul 30 15:49:49.209788 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.209657 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-home\") pod \"1df3097c-87ee-4819-bf01-810b4414194c\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " Jul 30 15:49:49.209788 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.209705 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-model-cache\") pod \"1df3097c-87ee-4819-bf01-810b4414194c\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " Jul 30 15:49:49.209788 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.209737 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-tmp-dir\") pod \"1df3097c-87ee-4819-bf01-810b4414194c\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " Jul 30 15:49:49.209788 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.209786 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-dshm\") pod \"1df3097c-87ee-4819-bf01-810b4414194c\" (UID: \"1df3097c-87ee-4819-bf01-810b4414194c\") " Jul 30 15:49:49.210062 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.210010 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1df3097c-87ee-4819-bf01-810b4414194c" (UID: "1df3097c-87ee-4819-bf01-810b4414194c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:49.210062 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.210022 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-home" (OuterVolumeSpecName: "home") pod "1df3097c-87ee-4819-bf01-810b4414194c" (UID: "1df3097c-87ee-4819-bf01-810b4414194c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:49.210186 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.210104 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1df3097c-87ee-4819-bf01-810b4414194c" (UID: "1df3097c-87ee-4819-bf01-810b4414194c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:49.210229 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.210183 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-model-cache\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:49.210229 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.210199 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-home\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:49.212281 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.212242 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1df3097c-87ee-4819-bf01-810b4414194c-kube-api-access-6wqwp" (OuterVolumeSpecName: "kube-api-access-6wqwp") pod "1df3097c-87ee-4819-bf01-810b4414194c" (UID: "1df3097c-87ee-4819-bf01-810b4414194c"). InnerVolumeSpecName "kube-api-access-6wqwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:49:49.219487 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.212778 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1df3097c-87ee-4819-bf01-810b4414194c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1df3097c-87ee-4819-bf01-810b4414194c" (UID: "1df3097c-87ee-4819-bf01-810b4414194c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:49:49.220262 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.220234 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-dshm" (OuterVolumeSpecName: "dshm") pod "1df3097c-87ee-4819-bf01-810b4414194c" (UID: "1df3097c-87ee-4819-bf01-810b4414194c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:49:49.310729 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.310692 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1df3097c-87ee-4819-bf01-810b4414194c-tls-certs\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:49.310729 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.310725 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wqwp\" (UniqueName: \"kubernetes.io/projected/1df3097c-87ee-4819-bf01-810b4414194c-kube-api-access-6wqwp\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:49.310729 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.310736 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-tmp-dir\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:49.310974 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.310744 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1df3097c-87ee-4819-bf01-810b4414194c-dshm\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:49:49.491215 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.491185 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="559a3ab7-c752-48ef-bcdd-9ab454d37f84" path="/var/lib/kubelet/pods/559a3ab7-c752-48ef-bcdd-9ab454d37f84/volumes" Jul 30 15:49:49.963504 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.963411 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" Jul 30 15:49:49.963504 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.963447 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2" event={"ID":"1df3097c-87ee-4819-bf01-810b4414194c","Type":"ContainerDied","Data":"44ded4c1652fddd65dbd7725ca46c48ddfe68b743fe5292d56a51fc6e8291c1e"} Jul 30 15:49:49.963760 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.963514 2564 scope.go:117] "RemoveContainer" containerID="e90fe9e99e026f617985aff77bc9857bf4a635796fc81ca3179dae1c7b9d3e4c" Jul 30 15:49:49.981243 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.981213 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2"] Jul 30 15:49:49.987235 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:49.987210 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-2b1f1f/rate-test-model-2b1f1f-kserve-855cff9c99-vvlx2"] Jul 30 15:49:51.494556 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:51.494502 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1df3097c-87ee-4819-bf01-810b4414194c" path="/var/lib/kubelet/pods/1df3097c-87ee-4819-bf01-810b4414194c/volumes" Jul 30 15:49:59.502150 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.502111 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh"] Jul 30 15:49:59.502642 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.502628 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1df3097c-87ee-4819-bf01-810b4414194c" containerName="main" Jul 30 15:49:59.502695 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.502644 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="1df3097c-87ee-4819-bf01-810b4414194c" containerName="main" Jul 30 15:49:59.502695 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.502658 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="559a3ab7-c752-48ef-bcdd-9ab454d37f84" containerName="test-curl-7548-139703280859152" Jul 30 15:49:59.502695 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.502668 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="559a3ab7-c752-48ef-bcdd-9ab454d37f84" containerName="test-curl-7548-139703280859152" Jul 30 15:49:59.502788 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.502742 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="559a3ab7-c752-48ef-bcdd-9ab454d37f84" containerName="test-curl-7548-139703280859152" Jul 30 15:49:59.502788 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.502758 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="1df3097c-87ee-4819-bf01-810b4414194c" containerName="main" Jul 30 15:49:59.505893 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.505879 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:49:59.507693 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.507667 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-10fa8b-dockercfg-6rnr4\"" Jul 30 15:49:59.507871 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.507855 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-10fa8b\"" Jul 30 15:49:59.514493 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.514443 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh"] Jul 30 15:49:59.540033 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.540004 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st"] Jul 30 15:49:59.543806 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.543787 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:49:59.555605 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.555578 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st"] Jul 30 15:49:59.613899 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.613869 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae41500-26e4-49c8-9ff4-3551726f5c44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st\" (UID: \"bae41500-26e4-49c8-9ff4-3551726f5c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:49:59.614096 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.613931 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgxcz\" (UniqueName: \"kubernetes.io/projected/12151cb2-ac6a-4383-945e-b33bf54d5130-kube-api-access-pgxcz\") pod \"payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh\" (UID: \"12151cb2-ac6a-4383-945e-b33bf54d5130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:49:59.614096 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.614015 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqlb4\" (UniqueName: \"kubernetes.io/projected/bae41500-26e4-49c8-9ff4-3551726f5c44-kube-api-access-zqlb4\") pod \"payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st\" (UID: \"bae41500-26e4-49c8-9ff4-3551726f5c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:49:59.614199 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.614097 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12151cb2-ac6a-4383-945e-b33bf54d5130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh\" (UID: \"12151cb2-ac6a-4383-945e-b33bf54d5130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:49:59.715449 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.715410 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12151cb2-ac6a-4383-945e-b33bf54d5130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh\" (UID: \"12151cb2-ac6a-4383-945e-b33bf54d5130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:49:59.715630 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.715491 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae41500-26e4-49c8-9ff4-3551726f5c44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st\" (UID: \"bae41500-26e4-49c8-9ff4-3551726f5c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:49:59.715630 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.715540 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgxcz\" (UniqueName: \"kubernetes.io/projected/12151cb2-ac6a-4383-945e-b33bf54d5130-kube-api-access-pgxcz\") pod \"payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh\" (UID: \"12151cb2-ac6a-4383-945e-b33bf54d5130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:49:59.715630 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.715575 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqlb4\" (UniqueName: \"kubernetes.io/projected/bae41500-26e4-49c8-9ff4-3551726f5c44-kube-api-access-zqlb4\") pod \"payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st\" (UID: \"bae41500-26e4-49c8-9ff4-3551726f5c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:49:59.716241 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.716216 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae41500-26e4-49c8-9ff4-3551726f5c44-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st\" (UID: \"bae41500-26e4-49c8-9ff4-3551726f5c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:49:59.716348 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.716274 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12151cb2-ac6a-4383-945e-b33bf54d5130-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh\" (UID: \"12151cb2-ac6a-4383-945e-b33bf54d5130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:49:59.723474 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.723437 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgxcz\" (UniqueName: \"kubernetes.io/projected/12151cb2-ac6a-4383-945e-b33bf54d5130-kube-api-access-pgxcz\") pod \"payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh\" (UID: \"12151cb2-ac6a-4383-945e-b33bf54d5130\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:49:59.723629 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.723611 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqlb4\" (UniqueName: \"kubernetes.io/projected/bae41500-26e4-49c8-9ff4-3551726f5c44-kube-api-access-zqlb4\") pod \"payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st\" (UID: \"bae41500-26e4-49c8-9ff4-3551726f5c44\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:49:59.823598 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.823481 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:49:59.855753 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.855709 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:49:59.987708 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:49:59.987671 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh"] Jul 30 15:49:59.989553 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:49:59.989521 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12151cb2_ac6a_4383_945e_b33bf54d5130.slice/crio-ad08eddc591147529a1929367ca53f66e6244c62c85d000e8a5c1d81c75ab09f WatchSource:0}: Error finding container ad08eddc591147529a1929367ca53f66e6244c62c85d000e8a5c1d81c75ab09f: Status 404 returned error can't find the container with id ad08eddc591147529a1929367ca53f66e6244c62c85d000e8a5c1d81c75ab09f Jul 30 15:50:00.028804 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:00.028769 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st"] Jul 30 15:50:00.031335 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:50:00.031301 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae41500_26e4_49c8_9ff4_3551726f5c44.slice/crio-04d79c705649b9f03d5dd28f5720addf053e9936c6c79a6e1af8273cd8b93a9e WatchSource:0}: Error finding container 04d79c705649b9f03d5dd28f5720addf053e9936c6c79a6e1af8273cd8b93a9e: Status 404 returned error can't find the container with id 04d79c705649b9f03d5dd28f5720addf053e9936c6c79a6e1af8273cd8b93a9e Jul 30 15:50:00.032642 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:00.032607 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" event={"ID":"12151cb2-ac6a-4383-945e-b33bf54d5130","Type":"ContainerStarted","Data":"ad08eddc591147529a1929367ca53f66e6244c62c85d000e8a5c1d81c75ab09f"} Jul 30 15:50:01.039955 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:01.039919 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" event={"ID":"12151cb2-ac6a-4383-945e-b33bf54d5130","Type":"ContainerStarted","Data":"a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff"} Jul 30 15:50:01.040409 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:01.040109 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:50:01.041653 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:01.041629 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" event={"ID":"bae41500-26e4-49c8-9ff4-3551726f5c44","Type":"ContainerStarted","Data":"04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3"} Jul 30 15:50:01.041782 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:01.041674 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" event={"ID":"bae41500-26e4-49c8-9ff4-3551726f5c44","Type":"ContainerStarted","Data":"04d79c705649b9f03d5dd28f5720addf053e9936c6c79a6e1af8273cd8b93a9e"} Jul 30 15:50:01.041782 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:01.041705 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:50:01.057581 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:01.057536 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" podStartSLOduration=2.057520892 podStartE2EDuration="2.057520892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:50:01.055226197 +0000 UTC m=+3728.038509534" watchObservedRunningTime="2026-07-30 15:50:01.057520892 +0000 UTC m=+3728.040804229" Jul 30 15:50:01.071183 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:01.071125 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" podStartSLOduration=2.071108407 podStartE2EDuration="2.071108407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:49:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:50:01.07057013 +0000 UTC m=+3728.053853476" watchObservedRunningTime="2026-07-30 15:50:01.071108407 +0000 UTC m=+3728.054391745" Jul 30 15:50:12.050157 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:12.050118 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:50:12.050663 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:50:12.050237 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:51:18.924970 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.924870 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x"] Jul 30 15:51:18.928918 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.928896 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:18.931678 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.931655 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-10fa8b\"/\"openshift-service-ca.crt\"" Jul 30 15:51:18.931678 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.931664 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-10fa8b\"/\"kube-root-ca.crt\"" Jul 30 15:51:18.931678 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.931654 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-10fa8b\"/\"default-dockercfg-x75kf\"" Jul 30 15:51:18.931921 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.931703 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-10fa8b\"/\"ipp-route-10fa8b-kserve-self-signed-certs\"" Jul 30 15:51:18.940009 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.939983 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x"] Jul 30 15:51:18.988940 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.988892 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-dshm\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:18.989252 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.988943 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-model-cache\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:18.989252 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.989003 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-home\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:18.989252 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.989035 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-tmp-dir\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:18.989252 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.989198 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w5tr\" (UniqueName: \"kubernetes.io/projected/f7e70877-1121-44c7-af64-c0f6028aac9d-kube-api-access-7w5tr\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:18.989252 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:18.989243 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e70877-1121-44c7-af64-c0f6028aac9d-tls-certs\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.090395 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.090350 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-dshm\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.090395 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.090399 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-model-cache\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.090665 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.090424 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-home\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.090665 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.090446 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-tmp-dir\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.090665 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.090524 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w5tr\" (UniqueName: \"kubernetes.io/projected/f7e70877-1121-44c7-af64-c0f6028aac9d-kube-api-access-7w5tr\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.090665 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.090559 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e70877-1121-44c7-af64-c0f6028aac9d-tls-certs\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.090936 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.090913 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-home\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.091052 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.090942 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-model-cache\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.091289 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.091267 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-tmp-dir\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.093003 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.092979 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-dshm\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.093331 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.093314 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e70877-1121-44c7-af64-c0f6028aac9d-tls-certs\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.101046 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.101021 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w5tr\" (UniqueName: \"kubernetes.io/projected/f7e70877-1121-44c7-af64-c0f6028aac9d-kube-api-access-7w5tr\") pod \"ipp-route-10fa8b-kserve-78bc884cf5-s9s5x\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.239036 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.238984 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:19.391276 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.391244 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x"] Jul 30 15:51:19.393167 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:51:19.393134 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7e70877_1121_44c7_af64_c0f6028aac9d.slice/crio-c9815cdc92f38e602daf855e81cf8b1581c39212b6594427ec49e61c14da2ad6 WatchSource:0}: Error finding container c9815cdc92f38e602daf855e81cf8b1581c39212b6594427ec49e61c14da2ad6: Status 404 returned error can't find the container with id c9815cdc92f38e602daf855e81cf8b1581c39212b6594427ec49e61c14da2ad6 Jul 30 15:51:19.532283 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.532181 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" event={"ID":"f7e70877-1121-44c7-af64-c0f6028aac9d","Type":"ContainerStarted","Data":"352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f"} Jul 30 15:51:19.532283 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.532223 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" event={"ID":"f7e70877-1121-44c7-af64-c0f6028aac9d","Type":"ContainerStarted","Data":"c9815cdc92f38e602daf855e81cf8b1581c39212b6594427ec49e61c14da2ad6"} Jul 30 15:51:19.550630 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:19.550581 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" podStartSLOduration=1.550564036 podStartE2EDuration="1.550564036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-30 15:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:51:19.549666952 +0000 UTC m=+3806.532950311" watchObservedRunningTime="2026-07-30 15:51:19.550564036 +0000 UTC m=+3806.533847372" Jul 30 15:51:29.239113 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:29.239052 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:29.239510 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:29.239141 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:29.251893 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:29.251869 2564 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:51:29.605686 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:51:29.605597 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:52:04.553753 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:52:04.553725 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh_12151cb2-ac6a-4383-945e-b33bf54d5130/payload-processing/0.log" Jul 30 15:52:54.005095 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:52:54.005043 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:52:54.030740 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:52:54.030710 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-6r57d_b76f0754-2727-41b2-9656-5bbca581adde/ovn-acl-logging/0.log" Jul 30 15:53:06.454012 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.453977 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh"] Jul 30 15:53:06.454431 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.454199 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" podUID="12151cb2-ac6a-4383-945e-b33bf54d5130" containerName="payload-processing" containerID="cri-o://a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff" gracePeriod=30 Jul 30 15:53:06.461292 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.461260 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st"] Jul 30 15:53:06.461525 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.461481 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" podUID="bae41500-26e4-49c8-9ff4-3551726f5c44" containerName="payload-pre-processing" containerID="cri-o://04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3" gracePeriod=30 Jul 30 15:53:06.742176 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.742148 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:53:06.761982 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.761952 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12151cb2-ac6a-4383-945e-b33bf54d5130-plugins-config-volume\") pod \"12151cb2-ac6a-4383-945e-b33bf54d5130\" (UID: \"12151cb2-ac6a-4383-945e-b33bf54d5130\") " Jul 30 15:53:06.762139 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.762039 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgxcz\" (UniqueName: \"kubernetes.io/projected/12151cb2-ac6a-4383-945e-b33bf54d5130-kube-api-access-pgxcz\") pod \"12151cb2-ac6a-4383-945e-b33bf54d5130\" (UID: \"12151cb2-ac6a-4383-945e-b33bf54d5130\") " Jul 30 15:53:06.762472 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.762431 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12151cb2-ac6a-4383-945e-b33bf54d5130-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "12151cb2-ac6a-4383-945e-b33bf54d5130" (UID: "12151cb2-ac6a-4383-945e-b33bf54d5130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:53:06.766121 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.766091 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12151cb2-ac6a-4383-945e-b33bf54d5130-kube-api-access-pgxcz" (OuterVolumeSpecName: "kube-api-access-pgxcz") pod "12151cb2-ac6a-4383-945e-b33bf54d5130" (UID: "12151cb2-ac6a-4383-945e-b33bf54d5130"). InnerVolumeSpecName "kube-api-access-pgxcz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:53:06.772165 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.772141 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:53:06.863385 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.863352 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae41500-26e4-49c8-9ff4-3551726f5c44-plugins-config-volume\") pod \"bae41500-26e4-49c8-9ff4-3551726f5c44\" (UID: \"bae41500-26e4-49c8-9ff4-3551726f5c44\") " Jul 30 15:53:06.863587 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.863557 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqlb4\" (UniqueName: \"kubernetes.io/projected/bae41500-26e4-49c8-9ff4-3551726f5c44-kube-api-access-zqlb4\") pod \"bae41500-26e4-49c8-9ff4-3551726f5c44\" (UID: \"bae41500-26e4-49c8-9ff4-3551726f5c44\") " Jul 30 15:53:06.863796 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.863779 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgxcz\" (UniqueName: \"kubernetes.io/projected/12151cb2-ac6a-4383-945e-b33bf54d5130-kube-api-access-pgxcz\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:06.863865 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.863795 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12151cb2-ac6a-4383-945e-b33bf54d5130-plugins-config-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:06.863865 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.863778 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae41500-26e4-49c8-9ff4-3551726f5c44-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bae41500-26e4-49c8-9ff4-3551726f5c44" (UID: "bae41500-26e4-49c8-9ff4-3551726f5c44"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:53:06.866062 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.866027 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae41500-26e4-49c8-9ff4-3551726f5c44-kube-api-access-zqlb4" (OuterVolumeSpecName: "kube-api-access-zqlb4") pod "bae41500-26e4-49c8-9ff4-3551726f5c44" (UID: "bae41500-26e4-49c8-9ff4-3551726f5c44"). InnerVolumeSpecName "kube-api-access-zqlb4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:53:06.964954 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.964848 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqlb4\" (UniqueName: \"kubernetes.io/projected/bae41500-26e4-49c8-9ff4-3551726f5c44-kube-api-access-zqlb4\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:06.964954 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:06.964895 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bae41500-26e4-49c8-9ff4-3551726f5c44-plugins-config-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:07.235034 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.234998 2564 generic.go:356] "Generic (PLEG): container finished" podID="12151cb2-ac6a-4383-945e-b33bf54d5130" containerID="a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff" exitCode=0 Jul 30 15:53:07.235219 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.235050 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" Jul 30 15:53:07.235219 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.235109 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" event={"ID":"12151cb2-ac6a-4383-945e-b33bf54d5130","Type":"ContainerDied","Data":"a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff"} Jul 30 15:53:07.235219 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.235162 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh" event={"ID":"12151cb2-ac6a-4383-945e-b33bf54d5130","Type":"ContainerDied","Data":"ad08eddc591147529a1929367ca53f66e6244c62c85d000e8a5c1d81c75ab09f"} Jul 30 15:53:07.235219 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.235184 2564 scope.go:117] "RemoveContainer" containerID="a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff" Jul 30 15:53:07.236487 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.236470 2564 generic.go:356] "Generic (PLEG): container finished" podID="bae41500-26e4-49c8-9ff4-3551726f5c44" containerID="04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3" exitCode=0 Jul 30 15:53:07.236585 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.236498 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" event={"ID":"bae41500-26e4-49c8-9ff4-3551726f5c44","Type":"ContainerDied","Data":"04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3"} Jul 30 15:53:07.236585 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.236523 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" Jul 30 15:53:07.236585 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.236560 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st" event={"ID":"bae41500-26e4-49c8-9ff4-3551726f5c44","Type":"ContainerDied","Data":"04d79c705649b9f03d5dd28f5720addf053e9936c6c79a6e1af8273cd8b93a9e"} Jul 30 15:53:07.245856 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.245839 2564 scope.go:117] "RemoveContainer" containerID="a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff" Jul 30 15:53:07.246162 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:53:07.246137 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff\": container with ID starting with a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff not found: ID does not exist" containerID="a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff" Jul 30 15:53:07.246290 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.246168 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff"} err="failed to get container status \"a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff\": rpc error: code = NotFound desc = could not find container \"a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff\": container with ID starting with a1c43f10256cb8f72476b07268522092d90299953f668b2da968fc7512963eff not found: ID does not exist" Jul 30 15:53:07.246290 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.246183 2564 scope.go:117] "RemoveContainer" containerID="04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3" Jul 30 15:53:07.255123 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.255105 2564 scope.go:117] "RemoveContainer" containerID="04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3" Jul 30 15:53:07.255390 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:53:07.255369 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3\": container with ID starting with 04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3 not found: ID does not exist" containerID="04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3" Jul 30 15:53:07.255440 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.255399 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3"} err="failed to get container status \"04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3\": rpc error: code = NotFound desc = could not find container \"04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3\": container with ID starting with 04c700f42b8c95ba65e467deea2482ff85ecb72ae592a5d298d38a5c9ff2acb3 not found: ID does not exist" Jul 30 15:53:07.263730 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.263705 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh"] Jul 30 15:53:07.271063 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.271037 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-10fa8b-76d4b9589d-4vzdh"] Jul 30 15:53:07.281130 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.281093 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st"] Jul 30 15:53:07.284246 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.284220 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-10fa8b-5b95fd84d4-nd8st"] Jul 30 15:53:07.492558 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.492475 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12151cb2-ac6a-4383-945e-b33bf54d5130" path="/var/lib/kubelet/pods/12151cb2-ac6a-4383-945e-b33bf54d5130/volumes" Jul 30 15:53:07.493061 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:07.493040 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bae41500-26e4-49c8-9ff4-3551726f5c44" path="/var/lib/kubelet/pods/bae41500-26e4-49c8-9ff4-3551726f5c44/volumes" Jul 30 15:53:12.001615 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:12.001571 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x"] Jul 30 15:53:27.021563 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.021501 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" podUID="f7e70877-1121-44c7-af64-c0f6028aac9d" containerName="main" containerID="cri-o://352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f" gracePeriod=45 Jul 30 15:53:27.295043 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.295011 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:53:27.361369 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.361334 2564 generic.go:356] "Generic (PLEG): container finished" podID="f7e70877-1121-44c7-af64-c0f6028aac9d" containerID="352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f" exitCode=0 Jul 30 15:53:27.361560 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.361380 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" Jul 30 15:53:27.361560 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.361420 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" event={"ID":"f7e70877-1121-44c7-af64-c0f6028aac9d","Type":"ContainerDied","Data":"352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f"} Jul 30 15:53:27.361560 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.361466 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x" event={"ID":"f7e70877-1121-44c7-af64-c0f6028aac9d","Type":"ContainerDied","Data":"c9815cdc92f38e602daf855e81cf8b1581c39212b6594427ec49e61c14da2ad6"} Jul 30 15:53:27.361560 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.361493 2564 scope.go:117] "RemoveContainer" containerID="352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f" Jul 30 15:53:27.366200 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366178 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-dshm\") pod \"f7e70877-1121-44c7-af64-c0f6028aac9d\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " Jul 30 15:53:27.366331 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366268 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-tmp-dir\") pod \"f7e70877-1121-44c7-af64-c0f6028aac9d\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " Jul 30 15:53:27.366331 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366295 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e70877-1121-44c7-af64-c0f6028aac9d-tls-certs\") pod \"f7e70877-1121-44c7-af64-c0f6028aac9d\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " Jul 30 15:53:27.366331 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366317 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-home\") pod \"f7e70877-1121-44c7-af64-c0f6028aac9d\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " Jul 30 15:53:27.366503 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366356 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-model-cache\") pod \"f7e70877-1121-44c7-af64-c0f6028aac9d\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " Jul 30 15:53:27.366503 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366384 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w5tr\" (UniqueName: \"kubernetes.io/projected/f7e70877-1121-44c7-af64-c0f6028aac9d-kube-api-access-7w5tr\") pod \"f7e70877-1121-44c7-af64-c0f6028aac9d\" (UID: \"f7e70877-1121-44c7-af64-c0f6028aac9d\") " Jul 30 15:53:27.366612 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366551 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f7e70877-1121-44c7-af64-c0f6028aac9d" (UID: "f7e70877-1121-44c7-af64-c0f6028aac9d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:53:27.366720 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366692 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-model-cache" (OuterVolumeSpecName: "model-cache") pod "f7e70877-1121-44c7-af64-c0f6028aac9d" (UID: "f7e70877-1121-44c7-af64-c0f6028aac9d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:53:27.366823 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366805 2564 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-tmp-dir\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:27.366885 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366831 2564 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-model-cache\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:27.366963 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.366935 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-home" (OuterVolumeSpecName: "home") pod "f7e70877-1121-44c7-af64-c0f6028aac9d" (UID: "f7e70877-1121-44c7-af64-c0f6028aac9d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:53:27.368533 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.368508 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-dshm" (OuterVolumeSpecName: "dshm") pod "f7e70877-1121-44c7-af64-c0f6028aac9d" (UID: "f7e70877-1121-44c7-af64-c0f6028aac9d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 30 15:53:27.368649 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.368552 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7e70877-1121-44c7-af64-c0f6028aac9d-kube-api-access-7w5tr" (OuterVolumeSpecName: "kube-api-access-7w5tr") pod "f7e70877-1121-44c7-af64-c0f6028aac9d" (UID: "f7e70877-1121-44c7-af64-c0f6028aac9d"). InnerVolumeSpecName "kube-api-access-7w5tr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:53:27.368649 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.368617 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7e70877-1121-44c7-af64-c0f6028aac9d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f7e70877-1121-44c7-af64-c0f6028aac9d" (UID: "f7e70877-1121-44c7-af64-c0f6028aac9d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 30 15:53:27.371622 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.371602 2564 scope.go:117] "RemoveContainer" containerID="352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f" Jul 30 15:53:27.371912 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:53:27.371890 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f\": container with ID starting with 352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f not found: ID does not exist" containerID="352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f" Jul 30 15:53:27.372012 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.371917 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f"} err="failed to get container status \"352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f\": rpc error: code = NotFound desc = could not find container \"352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f\": container with ID starting with 352258fbd25cf46c863517800b835efa94ab9a2760b7b76c1b634d622c44586f not found: ID does not exist" Jul 30 15:53:27.467889 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.467856 2564 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-dshm\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:27.467889 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.467886 2564 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7e70877-1121-44c7-af64-c0f6028aac9d-tls-certs\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:27.467889 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.467895 2564 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7e70877-1121-44c7-af64-c0f6028aac9d-home\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:27.467889 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.467902 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w5tr\" (UniqueName: \"kubernetes.io/projected/f7e70877-1121-44c7-af64-c0f6028aac9d-kube-api-access-7w5tr\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:53:27.685111 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.685004 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x"] Jul 30 15:53:27.691498 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:27.691461 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-10fa8b/ipp-route-10fa8b-kserve-78bc884cf5-s9s5x"] Jul 30 15:53:29.491387 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:53:29.491350 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7e70877-1121-44c7-af64-c0f6028aac9d" path="/var/lib/kubelet/pods/f7e70877-1121-44c7-af64-c0f6028aac9d/volumes" Jul 30 15:54:42.804900 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:42.804854 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h"] Jul 30 15:54:42.805515 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:42.805141 2564 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" podUID="fa351034-712c-445b-8bc0-7483a473997d" containerName="payload-pre-processing" containerID="cri-o://7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b" gracePeriod=30 Jul 30 15:54:43.075827 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.075794 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:54:43.218132 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.218066 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rmhf9\" (UniqueName: \"kubernetes.io/projected/fa351034-712c-445b-8bc0-7483a473997d-kube-api-access-rmhf9\") pod \"fa351034-712c-445b-8bc0-7483a473997d\" (UID: \"fa351034-712c-445b-8bc0-7483a473997d\") " Jul 30 15:54:43.218308 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.218216 2564 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa351034-712c-445b-8bc0-7483a473997d-plugins-config-volume\") pod \"fa351034-712c-445b-8bc0-7483a473997d\" (UID: \"fa351034-712c-445b-8bc0-7483a473997d\") " Jul 30 15:54:43.218653 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.218629 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa351034-712c-445b-8bc0-7483a473997d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa351034-712c-445b-8bc0-7483a473997d" (UID: "fa351034-712c-445b-8bc0-7483a473997d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 30 15:54:43.220246 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.220220 2564 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa351034-712c-445b-8bc0-7483a473997d-kube-api-access-rmhf9" (OuterVolumeSpecName: "kube-api-access-rmhf9") pod "fa351034-712c-445b-8bc0-7483a473997d" (UID: "fa351034-712c-445b-8bc0-7483a473997d"). InnerVolumeSpecName "kube-api-access-rmhf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 30 15:54:43.320053 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.319965 2564 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa351034-712c-445b-8bc0-7483a473997d-plugins-config-volume\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:54:43.320053 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.319995 2564 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmhf9\" (UniqueName: \"kubernetes.io/projected/fa351034-712c-445b-8bc0-7483a473997d-kube-api-access-rmhf9\") on node \"ip-10-0-130-38.ec2.internal\" DevicePath \"\"" Jul 30 15:54:43.850133 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.850065 2564 generic.go:356] "Generic (PLEG): container finished" podID="fa351034-712c-445b-8bc0-7483a473997d" containerID="7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b" exitCode=0 Jul 30 15:54:43.850639 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.850213 2564 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" Jul 30 15:54:43.850639 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.850307 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" event={"ID":"fa351034-712c-445b-8bc0-7483a473997d","Type":"ContainerDied","Data":"7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b"} Jul 30 15:54:43.850639 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.850367 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h" event={"ID":"fa351034-712c-445b-8bc0-7483a473997d","Type":"ContainerDied","Data":"574b813f05ceceb0abba334924e39ba6669794e2acecfa9f36049d2f27ff05a8"} Jul 30 15:54:43.850639 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.850401 2564 scope.go:117] "RemoveContainer" containerID="7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b" Jul 30 15:54:43.861271 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.861246 2564 scope.go:117] "RemoveContainer" containerID="7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b" Jul 30 15:54:43.861576 ip-10-0-130-38 kubenswrapper[2564]: E0730 15:54:43.861555 2564 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b\": container with ID starting with 7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b not found: ID does not exist" containerID="7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b" Jul 30 15:54:43.861647 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.861591 2564 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b"} err="failed to get container status \"7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b\": rpc error: code = NotFound desc = could not find container \"7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b\": container with ID starting with 7dd328f210ef73eefa08d97b8c1d0998ce56990459c04bd379b17e9fb7cd8e4b not found: ID does not exist" Jul 30 15:54:43.868496 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.868466 2564 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h"] Jul 30 15:54:43.873998 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:43.873965 2564 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-2b1f1f-7d988665bd-4649h"] Jul 30 15:54:45.491763 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:45.491731 2564 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa351034-712c-445b-8bc0-7483a473997d" path="/var/lib/kubelet/pods/fa351034-712c-445b-8bc0-7483a473997d/volumes" Jul 30 15:54:50.740918 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:50.740886 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55fb68957-4k4k6_3a2a5e88-5891-4e44-96cd-86aad43df808/authorino/0.log" Jul 30 15:54:56.412119 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:56.412053 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-46w6t_56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5/manager/0.log" Jul 30 15:54:56.541696 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:56.541655 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4zhf4_ffe366d3-2a9c-468f-90ce-b998446d83b8/manager/0.log" Jul 30 15:54:56.944516 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:56.944487 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-p8ncl_fc4d29fd-f24c-46b9-85cd-7bd355cd9467/manager/0.log" Jul 30 15:54:57.066553 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:57.066523 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh_0570a4c2-9c27-4de2-a4f5-06aac0174584/copy-manifests/0.log" Jul 30 15:54:57.080724 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:57.080696 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh_0570a4c2-9c27-4de2-a4f5-06aac0174584/manager/0.log" Jul 30 15:54:59.031423 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.031395 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4_0fb7a89f-57b3-41ab-bdd2-50ce14856956/util/0.log" Jul 30 15:54:59.038053 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.038006 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4_0fb7a89f-57b3-41ab-bdd2-50ce14856956/pull/0.log" Jul 30 15:54:59.043942 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.043923 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4_0fb7a89f-57b3-41ab-bdd2-50ce14856956/extract/0.log" Jul 30 15:54:59.160307 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.160259 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld_03c36b98-bfcf-41de-99ef-ad505c4fe8b8/extract/0.log" Jul 30 15:54:59.166883 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.166864 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld_03c36b98-bfcf-41de-99ef-ad505c4fe8b8/util/0.log" Jul 30 15:54:59.173696 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.173670 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld_03c36b98-bfcf-41de-99ef-ad505c4fe8b8/pull/0.log" Jul 30 15:54:59.288467 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.288387 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x_04c92fc9-14db-40f0-9932-68e69ae8c28c/util/0.log" Jul 30 15:54:59.295966 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.295930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x_04c92fc9-14db-40f0-9932-68e69ae8c28c/pull/0.log" Jul 30 15:54:59.303420 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.303401 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x_04c92fc9-14db-40f0-9932-68e69ae8c28c/extract/0.log" Jul 30 15:54:59.419702 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.419676 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk_d1a90ab8-22a9-4ecc-b853-2f587a7ae968/util/0.log" Jul 30 15:54:59.426864 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.426846 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk_d1a90ab8-22a9-4ecc-b853-2f587a7ae968/pull/0.log" Jul 30 15:54:59.434126 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.434108 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk_d1a90ab8-22a9-4ecc-b853-2f587a7ae968/extract/0.log" Jul 30 15:54:59.560688 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.560609 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55fb68957-4k4k6_3a2a5e88-5891-4e44-96cd-86aad43df808/authorino/0.log" Jul 30 15:54:59.687391 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.687358 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qp8pt_5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd/manager/0.log" Jul 30 15:54:59.815897 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.815823 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-hcd8m_9e5b60ac-42ef-4f44-8026-803a4bf045ba/manager/0.log" Jul 30 15:54:59.933421 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:54:59.933393 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lw6gk_e6f8872c-8d81-4c86-9ae5-e33d0935a23d/kuadrant-console-plugin/0.log" Jul 30 15:55:00.066483 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:00.066402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4mcl5_086a4804-a211-4335-ad43-a1c1dade9895/registry-server/0.log" Jul 30 15:55:00.205497 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:00.205470 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dxc52_207dd104-3c82-46b5-886d-4036d4c0cb6c/manager/0.log" Jul 30 15:55:00.458844 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:00.458760 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4vg2s_3a86e7cb-c0ab-4858-9507-577203b2cbeb/manager/0.log" Jul 30 15:55:00.840152 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:00.840118 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt_0e73061e-8050-4b5e-b10e-33bd1e2e9454/istio-proxy/0.log" Jul 30 15:55:01.096343 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:01.096224 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c4bbddff-cfr6h_5ceb2d7e-a261-475c-a09d-30a6b77dbe16/kube-auth-proxy/0.log" Jul 30 15:55:01.345166 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:01.345128 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9sfjv_7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61/istio-proxy/1.log" Jul 30 15:55:01.470967 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:01.470887 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wj2pk_1ffefa41-b18b-4eee-9c48-6f0bba8f2f18/payload-pre-processing/0.log" Jul 30 15:55:02.107914 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:02.107885 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zdjrr_768617bf-fad6-475c-b760-5223bed08502/main/0.log" Jul 30 15:55:02.358116 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:02.357988 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-5hzdz_2a720ed2-9b7f-4615-a3be-6c9437b47f54/main/0.log" Jul 30 15:55:02.609620 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:02.609510 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-j9755_91f68cda-c1d2-455c-8e3a-176eeda73b35/main/0.log" Jul 30 15:55:11.430865 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:11.430834 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p86cc_0048d797-3821-4f51-92ed-84cb6b974158/global-pull-secret-syncer/0.log" Jul 30 15:55:11.518453 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:11.518406 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-8v5bk_cf992175-6c90-49cd-ae87-1f07a3ecdcef/konnectivity-agent/0.log" Jul 30 15:55:11.585632 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:11.585602 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-38.ec2.internal_b654a953333e0c5fd335382e9fed72ea/haproxy/0.log" Jul 30 15:55:16.222473 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.222402 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4_0fb7a89f-57b3-41ab-bdd2-50ce14856956/extract/0.log" Jul 30 15:55:16.266128 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.266083 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4_0fb7a89f-57b3-41ab-bdd2-50ce14856956/util/0.log" Jul 30 15:55:16.294776 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.294740 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759hwxw4_0fb7a89f-57b3-41ab-bdd2-50ce14856956/pull/0.log" Jul 30 15:55:16.330693 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.330666 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld_03c36b98-bfcf-41de-99ef-ad505c4fe8b8/extract/0.log" Jul 30 15:55:16.359465 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.359433 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld_03c36b98-bfcf-41de-99ef-ad505c4fe8b8/util/0.log" Jul 30 15:55:16.387091 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.387054 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0rvtld_03c36b98-bfcf-41de-99ef-ad505c4fe8b8/pull/0.log" Jul 30 15:55:16.419512 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.419461 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x_04c92fc9-14db-40f0-9932-68e69ae8c28c/extract/0.log" Jul 30 15:55:16.451608 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.451575 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x_04c92fc9-14db-40f0-9932-68e69ae8c28c/util/0.log" Jul 30 15:55:16.485385 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.485357 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73wsd5x_04c92fc9-14db-40f0-9932-68e69ae8c28c/pull/0.log" Jul 30 15:55:16.524445 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.524418 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk_d1a90ab8-22a9-4ecc-b853-2f587a7ae968/extract/0.log" Jul 30 15:55:16.553734 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.553708 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk_d1a90ab8-22a9-4ecc-b853-2f587a7ae968/util/0.log" Jul 30 15:55:16.586373 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.586333 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1w4nhk_d1a90ab8-22a9-4ecc-b853-2f587a7ae968/pull/0.log" Jul 30 15:55:16.995790 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:16.995761 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-55fb68957-4k4k6_3a2a5e88-5891-4e44-96cd-86aad43df808/authorino/0.log" Jul 30 15:55:17.049059 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:17.049030 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-qp8pt_5bd7d64b-df24-4133-b7ca-fa0f1ac4a2bd/manager/0.log" Jul 30 15:55:17.118802 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:17.118776 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-hcd8m_9e5b60ac-42ef-4f44-8026-803a4bf045ba/manager/0.log" Jul 30 15:55:17.160884 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:17.160860 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lw6gk_e6f8872c-8d81-4c86-9ae5-e33d0935a23d/kuadrant-console-plugin/0.log" Jul 30 15:55:17.218698 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:17.218660 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-4mcl5_086a4804-a211-4335-ad43-a1c1dade9895/registry-server/0.log" Jul 30 15:55:17.299464 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:17.299379 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-dxc52_207dd104-3c82-46b5-886d-4036d4c0cb6c/manager/0.log" Jul 30 15:55:17.405738 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:17.405707 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-4vg2s_3a86e7cb-c0ab-4858-9507-577203b2cbeb/manager/0.log" Jul 30 15:55:19.075606 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.075571 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be4a0dda-fbae-4e8e-a056-11e6d0b709b2/alertmanager/0.log" Jul 30 15:55:19.101754 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.101722 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be4a0dda-fbae-4e8e-a056-11e6d0b709b2/config-reloader/0.log" Jul 30 15:55:19.129387 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.129360 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be4a0dda-fbae-4e8e-a056-11e6d0b709b2/kube-rbac-proxy-web/0.log" Jul 30 15:55:19.161880 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.161851 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be4a0dda-fbae-4e8e-a056-11e6d0b709b2/kube-rbac-proxy/0.log" Jul 30 15:55:19.190513 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.190488 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be4a0dda-fbae-4e8e-a056-11e6d0b709b2/kube-rbac-proxy-metric/0.log" Jul 30 15:55:19.217633 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.217605 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be4a0dda-fbae-4e8e-a056-11e6d0b709b2/prom-label-proxy/0.log" Jul 30 15:55:19.246660 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.246627 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_be4a0dda-fbae-4e8e-a056-11e6d0b709b2/init-config-reloader/0.log" Jul 30 15:55:19.293418 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.293388 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-548b489cb8-2qq5d_31303ff1-ec5b-491b-8810-ea15f989b7c8/cluster-monitoring-operator/0.log" Jul 30 15:55:19.335976 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.335901 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-9szkx_0dcc3378-f0c7-4590-80d7-ef8061034f14/kube-state-metrics/0.log" Jul 30 15:55:19.361987 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.361963 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-9szkx_0dcc3378-f0c7-4590-80d7-ef8061034f14/kube-rbac-proxy-main/0.log" Jul 30 15:55:19.395000 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.394968 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56d9f944b4-9szkx_0dcc3378-f0c7-4590-80d7-ef8061034f14/kube-rbac-proxy-self/0.log" Jul 30 15:55:19.662981 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.662905 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtrc2_c11b1f6d-18f2-492e-9119-d58d7365342a/node-exporter/0.log" Jul 30 15:55:19.689101 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.689062 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtrc2_c11b1f6d-18f2-492e-9119-d58d7365342a/kube-rbac-proxy/0.log" Jul 30 15:55:19.716407 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:19.716376 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-qtrc2_c11b1f6d-18f2-492e-9119-d58d7365342a/init-textfile/0.log" Jul 30 15:55:20.008153 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.008124 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-47ks2_680eabbc-c97f-4554-979f-f8d88255f5f0/prometheus-operator/0.log" Jul 30 15:55:20.034568 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.034530 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-47ks2_680eabbc-c97f-4554-979f-f8d88255f5f0/kube-rbac-proxy/0.log" Jul 30 15:55:20.065951 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.065917 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-7f8d78f54b-rbnbf_514622fb-2dfc-4ac3-bad9-3e596a838b82/prometheus-operator-admission-webhook/0.log" Jul 30 15:55:20.102179 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.102151 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69d6989cf7-6hrpg_1a206462-f46b-4af4-9775-d45ffcd98de1/telemeter-client/0.log" Jul 30 15:55:20.132948 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.132922 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69d6989cf7-6hrpg_1a206462-f46b-4af4-9775-d45ffcd98de1/reload/0.log" Jul 30 15:55:20.164770 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.164746 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-69d6989cf7-6hrpg_1a206462-f46b-4af4-9775-d45ffcd98de1/kube-rbac-proxy/0.log" Jul 30 15:55:20.209223 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.209194 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b9d755bf9-k4c7h_89b12dec-2cf1-46a3-b49f-4618aaa3df6b/thanos-query/0.log" Jul 30 15:55:20.237239 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.237214 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b9d755bf9-k4c7h_89b12dec-2cf1-46a3-b49f-4618aaa3df6b/kube-rbac-proxy-web/0.log" Jul 30 15:55:20.262997 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.262930 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b9d755bf9-k4c7h_89b12dec-2cf1-46a3-b49f-4618aaa3df6b/kube-rbac-proxy/0.log" Jul 30 15:55:20.288413 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.288379 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b9d755bf9-k4c7h_89b12dec-2cf1-46a3-b49f-4618aaa3df6b/prom-label-proxy/0.log" Jul 30 15:55:20.323183 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.323157 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b9d755bf9-k4c7h_89b12dec-2cf1-46a3-b49f-4618aaa3df6b/kube-rbac-proxy-rules/0.log" Jul 30 15:55:20.374031 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:20.374002 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-7b9d755bf9-k4c7h_89b12dec-2cf1-46a3-b49f-4618aaa3df6b/kube-rbac-proxy-metrics/0.log" Jul 30 15:55:21.755990 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.755894 2564 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v"] Jul 30 15:55:21.756441 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756423 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12151cb2-ac6a-4383-945e-b33bf54d5130" containerName="payload-processing" Jul 30 15:55:21.756441 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756441 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="12151cb2-ac6a-4383-945e-b33bf54d5130" containerName="payload-processing" Jul 30 15:55:21.756522 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756452 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7e70877-1121-44c7-af64-c0f6028aac9d" containerName="main" Jul 30 15:55:21.756522 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756458 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7e70877-1121-44c7-af64-c0f6028aac9d" containerName="main" Jul 30 15:55:21.756522 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756476 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa351034-712c-445b-8bc0-7483a473997d" containerName="payload-pre-processing" Jul 30 15:55:21.756522 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756482 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa351034-712c-445b-8bc0-7483a473997d" containerName="payload-pre-processing" Jul 30 15:55:21.756522 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756499 2564 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bae41500-26e4-49c8-9ff4-3551726f5c44" containerName="payload-pre-processing" Jul 30 15:55:21.756522 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756505 2564 state_mem.go:107] "Deleted CPUSet assignment" podUID="bae41500-26e4-49c8-9ff4-3551726f5c44" containerName="payload-pre-processing" Jul 30 15:55:21.756706 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756568 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7e70877-1121-44c7-af64-c0f6028aac9d" containerName="main" Jul 30 15:55:21.756706 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756578 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="bae41500-26e4-49c8-9ff4-3551726f5c44" containerName="payload-pre-processing" Jul 30 15:55:21.756706 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756585 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa351034-712c-445b-8bc0-7483a473997d" containerName="payload-pre-processing" Jul 30 15:55:21.756706 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.756592 2564 memory_manager.go:368] "RemoveStaleState removing state" podUID="12151cb2-ac6a-4383-945e-b33bf54d5130" containerName="payload-processing" Jul 30 15:55:21.760168 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.760150 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.762483 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.762458 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ww6sr\"/\"openshift-service-ca.crt\"" Jul 30 15:55:21.762931 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.762909 2564 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ww6sr\"/\"kube-root-ca.crt\"" Jul 30 15:55:21.763031 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.762909 2564 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ww6sr\"/\"default-dockercfg-sp7jx\"" Jul 30 15:55:21.770564 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.770535 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v"] Jul 30 15:55:21.888259 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.888220 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9v7h\" (UniqueName: \"kubernetes.io/projected/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-kube-api-access-s9v7h\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.888521 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.888285 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-proc\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.888521 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.888344 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-sys\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.888521 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.888362 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-lib-modules\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.888521 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.888422 2564 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-podres\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989407 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989375 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-sys\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989407 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989411 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-lib-modules\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989619 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989438 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-podres\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989619 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989504 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9v7h\" (UniqueName: \"kubernetes.io/projected/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-kube-api-access-s9v7h\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989619 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989506 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-sys\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989619 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989579 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-lib-modules\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989619 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989585 2564 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-proc\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989619 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989617 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-proc\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.989810 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.989620 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-podres\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:21.999555 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:21.999529 2564 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9v7h\" (UniqueName: \"kubernetes.io/projected/b1bacfb4-5799-416e-b8ba-470c0e0aa9fe-kube-api-access-s9v7h\") pod \"perf-node-gather-daemonset-dlx4v\" (UID: \"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe\") " pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:22.071555 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:22.071480 2564 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:22.231642 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:22.231616 2564 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v"] Jul 30 15:55:22.233136 ip-10-0-130-38 kubenswrapper[2564]: W0730 15:55:22.233111 2564 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podb1bacfb4_5799_416e_b8ba_470c0e0aa9fe.slice/crio-f64682707fafd1e09792b6d2b141ae96ff63eb2aa4471e9741c4c8630f0b4d74 WatchSource:0}: Error finding container f64682707fafd1e09792b6d2b141ae96ff63eb2aa4471e9741c4c8630f0b4d74: Status 404 returned error can't find the container with id f64682707fafd1e09792b6d2b141ae96ff63eb2aa4471e9741c4c8630f0b4d74 Jul 30 15:55:22.786464 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:22.786437 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-86958f67c4-dsw84_ed54d4dd-e381-4203-aa71-5830b90ba9a6/console/0.log" Jul 30 15:55:23.093432 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:23.093349 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" event={"ID":"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe","Type":"ContainerStarted","Data":"3c4d4f89b1f4aa8ecd61845e5c8ce452a2da8a4fe2a9bb89d3077c5d692d8ded"} Jul 30 15:55:23.093432 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:23.093403 2564 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" event={"ID":"b1bacfb4-5799-416e-b8ba-470c0e0aa9fe","Type":"ContainerStarted","Data":"f64682707fafd1e09792b6d2b141ae96ff63eb2aa4471e9741c4c8630f0b4d74"} Jul 30 15:55:23.093632 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:23.093471 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:23.119445 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:23.119379 2564 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" podStartSLOduration=2.11935702 podStartE2EDuration="2.11935702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-30 15:55:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-30 15:55:23.11848124 +0000 UTC m=+4050.101764577" watchObservedRunningTime="2026-07-30 15:55:23.11935702 +0000 UTC m=+4050.102640358" Jul 30 15:55:24.530922 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:24.530892 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gx9sl_3e334195-bba1-46db-8e7b-b746dbb85ea8/dns/0.log" Jul 30 15:55:24.558456 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:24.558427 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-gx9sl_3e334195-bba1-46db-8e7b-b746dbb85ea8/kube-rbac-proxy/0.log" Jul 30 15:55:24.721993 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:24.721966 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dhkrf_36159941-b1b2-4d47-98ce-cf212bf196a2/dns-node-resolver/0.log" Jul 30 15:55:25.388061 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:25.388026 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cxskg_771dcee2-efa1-4503-9e75-16b904ed3df2/node-ca/0.log" Jul 30 15:55:26.446028 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:26.445997 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt_0e73061e-8050-4b5e-b10e-33bd1e2e9454/istio-proxy/0.log" Jul 30 15:55:26.615854 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:26.615823 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c4bbddff-cfr6h_5ceb2d7e-a261-475c-a09d-30a6b77dbe16/kube-auth-proxy/0.log" Jul 30 15:55:26.679917 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:26.679892 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9sfjv_7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61/istio-proxy/0.log" Jul 30 15:55:26.706227 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:26.706141 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9sfjv_7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61/istio-proxy/1.log" Jul 30 15:55:26.749343 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:26.749316 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wj2pk_1ffefa41-b18b-4eee-9c48-6f0bba8f2f18/payload-pre-processing/0.log" Jul 30 15:55:27.669057 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:27.669028 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2kjxw_99e1e1d4-1ab0-47fb-a4cd-a2d99d49644a/serve-healthcheck-canary/0.log" Jul 30 15:55:28.384657 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:28.384622 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8s2b_3a92369c-66e2-4c03-a22d-8ef9cc74231c/kube-rbac-proxy/0.log" Jul 30 15:55:28.408281 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:28.408252 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8s2b_3a92369c-66e2-4c03-a22d-8ef9cc74231c/exporter/0.log" Jul 30 15:55:28.440017 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:28.439986 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-f8s2b_3a92369c-66e2-4c03-a22d-8ef9cc74231c/extractor/0.log" Jul 30 15:55:29.110839 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:29.110812 2564 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ww6sr/perf-node-gather-daemonset-dlx4v" Jul 30 15:55:30.858697 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:30.858613 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-46w6t_56c5ef2f-c7be-4fa8-a7c5-d4eb631201a5/manager/0.log" Jul 30 15:55:31.089156 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:31.089129 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-4zhf4_ffe366d3-2a9c-468f-90ce-b998446d83b8/manager/0.log" Jul 30 15:55:31.406473 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:31.406444 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-p8ncl_fc4d29fd-f24c-46b9-85cd-7bd355cd9467/manager/0.log" Jul 30 15:55:31.514302 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:31.514267 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh_0570a4c2-9c27-4de2-a4f5-06aac0174584/manager/0.log" Jul 30 15:55:31.538043 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:31.537961 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6bdff7d7c7-6qlkh_0570a4c2-9c27-4de2-a4f5-06aac0174584/copy-manifests/0.log" Jul 30 15:55:31.940158 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:31.940060 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt_0e73061e-8050-4b5e-b10e-33bd1e2e9454/istio-proxy/0.log" Jul 30 15:55:32.093795 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:32.093767 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c4bbddff-cfr6h_5ceb2d7e-a261-475c-a09d-30a6b77dbe16/kube-auth-proxy/0.log" Jul 30 15:55:32.134513 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:32.134482 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9sfjv_7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61/istio-proxy/0.log" Jul 30 15:55:32.161717 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:32.161687 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9sfjv_7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61/istio-proxy/1.log" Jul 30 15:55:32.184543 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:32.184514 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wj2pk_1ffefa41-b18b-4eee-9c48-6f0bba8f2f18/payload-pre-processing/0.log" Jul 30 15:55:33.719970 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:33.719884 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-7k8mt_0e73061e-8050-4b5e-b10e-33bd1e2e9454/istio-proxy/0.log" Jul 30 15:55:33.883774 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:33.883741 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-59c4bbddff-cfr6h_5ceb2d7e-a261-475c-a09d-30a6b77dbe16/kube-auth-proxy/0.log" Jul 30 15:55:33.924165 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:33.924135 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9sfjv_7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61/istio-proxy/0.log" Jul 30 15:55:33.949778 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:33.949754 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-9sfjv_7f2f1253-4668-4d7e-9bf7-c8c2ac88cb61/istio-proxy/1.log" Jul 30 15:55:33.971889 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:33.971804 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-wj2pk_1ffefa41-b18b-4eee-9c48-6f0bba8f2f18/payload-pre-processing/0.log" Jul 30 15:55:37.857716 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:37.857681 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zpt7m_20224d9f-7519-4467-aacf-3a6d0675d45b/migrator/0.log" Jul 30 15:55:37.899469 ip-10-0-130-38 kubenswrapper[2564]: I0730 15:55:37.899436 2564 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-zpt7m_20224d9f-7519-4467-aacf-3a6d0675d45b/graceful-termination/0.log"