Jul 14 18:21:45.828593 ip-10-0-130-79 systemd[1]: Starting Kubernetes Kubelet... Jul 14 18:21:46.205199 ip-10-0-130-79 kubenswrapper[2569]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:46.205199 ip-10-0-130-79 kubenswrapper[2569]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 14 18:21:46.205199 ip-10-0-130-79 kubenswrapper[2569]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:46.205199 ip-10-0-130-79 kubenswrapper[2569]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 14 18:21:46.205199 ip-10-0-130-79 kubenswrapper[2569]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 14 18:21:46.206023 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.205225 2569 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 14 18:21:46.209371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209347 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:46.209371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209367 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:46.209371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209372 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209379 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209385 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209390 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209394 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209398 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209402 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209406 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209409 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209415 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209419 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209423 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209427 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209431 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209434 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209438 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209442 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209446 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209450 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:46.209578 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209454 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209458 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209462 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209467 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209471 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209474 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209478 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209482 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209486 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209490 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209495 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209499 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209503 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209507 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209512 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209517 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209521 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209527 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209531 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209537 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:46.210361 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209541 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209545 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209550 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209554 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209558 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209563 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209567 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209571 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209575 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209579 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209583 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209587 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209591 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209595 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209599 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209604 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209609 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209613 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209618 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209622 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:46.211144 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209626 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209631 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209635 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209639 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209643 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209647 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209652 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209657 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209661 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209665 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209671 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209676 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209681 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209685 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209690 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209694 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209699 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209703 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209709 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209714 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:46.211970 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209718 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209722 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209726 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209730 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209738 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209744 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209749 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209754 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209759 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209763 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209767 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209772 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209776 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209783 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209788 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209793 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209797 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.209801 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211055 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:46.212478 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211067 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211073 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211078 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211083 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211091 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211097 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211102 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211106 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211110 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211115 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211119 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211123 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211127 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211132 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211136 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211141 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211147 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211153 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211157 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:46.212943 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211161 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211166 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211174 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211178 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211183 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211187 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211192 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211196 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211200 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211204 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211208 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211212 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211217 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211222 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211227 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211232 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211237 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211241 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211246 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211269 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:46.213438 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211274 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211278 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211283 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211287 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211291 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211295 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211300 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211304 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211308 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211312 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211316 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211321 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211325 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211330 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211334 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211339 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211343 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211347 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211353 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211357 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:46.213911 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211361 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211365 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211370 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211374 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211378 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211382 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211386 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211391 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211395 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211399 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211411 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211419 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211424 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211428 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211432 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211436 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211440 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211447 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211451 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211455 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:46.214425 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211460 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211464 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211468 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211472 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211476 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211480 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211485 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211489 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211493 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211498 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211502 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211507 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211511 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211515 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211520 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211526 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211532 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211537 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.211542 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:46.214898 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212176 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212190 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212193 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212196 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212200 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212202 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212205 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212208 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212211 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212214 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212216 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212219 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212222 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212224 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212227 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212230 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212232 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212236 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212239 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212241 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:46.215406 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212244 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212247 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212249 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212264 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212267 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212270 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212273 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212275 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212278 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212281 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212284 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212286 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212289 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212295 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212299 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212302 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212305 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212308 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212311 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212314 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:46.215877 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212317 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212319 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212322 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212325 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212327 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212330 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212332 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212337 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212340 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212344 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212347 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212349 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212352 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212354 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212357 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212360 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212363 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212365 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212368 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:46.216368 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212370 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212373 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212376 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212378 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212381 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212384 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212387 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212390 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212393 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212396 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212398 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212401 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212404 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212406 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212409 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212411 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212414 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212416 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212419 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212422 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:46.216862 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212425 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212427 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212430 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212433 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212435 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212438 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212441 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212444 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212446 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212449 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212451 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212455 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212460 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212463 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212465 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212469 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212472 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212476 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212479 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:46.217371 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.212482 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212554 2569 flags.go:64] FLAG: --address="0.0.0.0" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212566 2569 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212577 2569 flags.go:64] FLAG: --anonymous-auth="true" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212583 2569 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212588 2569 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212591 2569 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212595 2569 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212600 2569 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212603 2569 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212607 2569 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212610 2569 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212614 2569 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212617 2569 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212620 2569 flags.go:64] FLAG: --cgroup-root="" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212623 2569 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212626 2569 flags.go:64] FLAG: --client-ca-file="" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212629 2569 flags.go:64] FLAG: --cloud-provider="external" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212632 2569 flags.go:64] FLAG: --cluster-dns="[]" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212636 2569 flags.go:64] FLAG: --cluster-domain="" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212639 2569 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212642 2569 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212645 2569 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212649 2569 flags.go:64] FLAG: --container-log-max-files="5" Jul 14 18:21:46.217823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212653 2569 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212656 2569 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212659 2569 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212663 2569 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212666 2569 flags.go:64] FLAG: --contention-profiling="false" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212669 2569 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212673 2569 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212677 2569 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212680 2569 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212684 2569 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212688 2569 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212691 2569 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212694 2569 flags.go:64] FLAG: --enable-load-reader="false" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212697 2569 flags.go:64] FLAG: --enable-server="true" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212701 2569 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212706 2569 flags.go:64] FLAG: --event-burst="100" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212709 2569 flags.go:64] FLAG: --event-qps="50" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212712 2569 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212715 2569 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212718 2569 flags.go:64] FLAG: --eviction-hard="" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212723 2569 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212726 2569 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212729 2569 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212732 2569 flags.go:64] FLAG: --eviction-soft="" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212735 2569 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 14 18:21:46.218457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212738 2569 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212741 2569 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212745 2569 flags.go:64] FLAG: --experimental-mounter-path="" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212748 2569 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212751 2569 flags.go:64] FLAG: --fail-swap-on="true" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212753 2569 flags.go:64] FLAG: --feature-gates="" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212757 2569 flags.go:64] FLAG: --file-check-frequency="20s" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212761 2569 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212764 2569 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212768 2569 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212771 2569 flags.go:64] FLAG: --healthz-port="10248" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212774 2569 flags.go:64] FLAG: --help="false" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212777 2569 flags.go:64] FLAG: --hostname-override="ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212780 2569 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212783 2569 flags.go:64] FLAG: --http-check-frequency="20s" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212786 2569 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212789 2569 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212793 2569 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212797 2569 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212800 2569 flags.go:64] FLAG: --image-service-endpoint="" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212803 2569 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212806 2569 flags.go:64] FLAG: --kube-api-burst="100" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212809 2569 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212812 2569 flags.go:64] FLAG: --kube-api-qps="50" Jul 14 18:21:46.219061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212815 2569 flags.go:64] FLAG: --kube-reserved="" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212817 2569 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212820 2569 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212823 2569 flags.go:64] FLAG: --kubelet-cgroups="" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212826 2569 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212829 2569 flags.go:64] FLAG: --lock-file="" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212832 2569 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212835 2569 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212838 2569 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212843 2569 flags.go:64] FLAG: --log-json-split-stream="false" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212846 2569 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212849 2569 flags.go:64] FLAG: --log-text-split-stream="false" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212852 2569 flags.go:64] FLAG: --logging-format="text" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212855 2569 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212858 2569 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212862 2569 flags.go:64] FLAG: --manifest-url="" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212865 2569 flags.go:64] FLAG: --manifest-url-header="" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212870 2569 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212873 2569 flags.go:64] FLAG: --max-open-files="1000000" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212877 2569 flags.go:64] FLAG: --max-pods="110" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212880 2569 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212883 2569 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212887 2569 flags.go:64] FLAG: --memory-manager-policy="None" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212890 2569 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212893 2569 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 14 18:21:46.219753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212896 2569 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212899 2569 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212908 2569 flags.go:64] FLAG: --node-status-max-images="50" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212911 2569 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212914 2569 flags.go:64] FLAG: --oom-score-adj="-999" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212918 2569 flags.go:64] FLAG: --pod-cidr="" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212921 2569 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212927 2569 flags.go:64] FLAG: --pod-manifest-path="" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212930 2569 flags.go:64] FLAG: --pod-max-pids="-1" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212933 2569 flags.go:64] FLAG: --pods-per-core="0" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212936 2569 flags.go:64] FLAG: --port="10250" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212939 2569 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212942 2569 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-065c39a9441434305" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212946 2569 flags.go:64] FLAG: --qos-reserved="" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212949 2569 flags.go:64] FLAG: --read-only-port="10255" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212952 2569 flags.go:64] FLAG: --register-node="true" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212956 2569 flags.go:64] FLAG: --register-with-taints="" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212960 2569 flags.go:64] FLAG: --registry-burst="10" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212963 2569 flags.go:64] FLAG: --registry-qps="5" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212966 2569 flags.go:64] FLAG: --reserved-cpus="" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212969 2569 flags.go:64] FLAG: --reserved-memory="" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212973 2569 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212976 2569 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212980 2569 flags.go:64] FLAG: --rotate-certificates="false" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212982 2569 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 14 18:21:46.220395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212985 2569 flags.go:64] FLAG: --runonce="false" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212988 2569 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212992 2569 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212995 2569 flags.go:64] FLAG: --seccomp-default="false" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.212997 2569 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213000 2569 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213004 2569 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213007 2569 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213010 2569 flags.go:64] FLAG: --storage-driver-password="root" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213013 2569 flags.go:64] FLAG: --storage-driver-secure="false" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213016 2569 flags.go:64] FLAG: --storage-driver-table="stats" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213019 2569 flags.go:64] FLAG: --storage-driver-user="root" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213022 2569 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213025 2569 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213028 2569 flags.go:64] FLAG: --system-cgroups="" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213031 2569 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213036 2569 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213039 2569 flags.go:64] FLAG: --tls-cert-file="" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213042 2569 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213046 2569 flags.go:64] FLAG: --tls-min-version="" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213049 2569 flags.go:64] FLAG: --tls-private-key-file="" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213052 2569 flags.go:64] FLAG: --topology-manager-policy="none" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213055 2569 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213058 2569 flags.go:64] FLAG: --topology-manager-scope="container" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213061 2569 flags.go:64] FLAG: --v="2" Jul 14 18:21:46.220992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213065 2569 flags.go:64] FLAG: --version="false" Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213069 2569 flags.go:64] FLAG: --vmodule="" Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213073 2569 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.213077 2569 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213221 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213227 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213230 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213233 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213236 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213239 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213242 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213245 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213248 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213271 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213274 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213277 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213282 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213286 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213295 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213298 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213301 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:46.221611 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213304 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213307 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213309 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213312 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213314 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213317 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213319 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213322 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213325 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213327 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213332 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213334 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213337 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213340 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213343 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213345 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213348 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213351 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213353 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213356 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:46.222151 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213358 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213361 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213364 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213366 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213369 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213372 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213374 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213378 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213381 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213384 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213392 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213395 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213397 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213400 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213403 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213406 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213409 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213411 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213414 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213416 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:46.222650 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213419 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213422 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213426 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213429 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213431 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213434 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213436 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213439 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213444 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213447 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213450 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213454 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213456 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213459 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213462 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213464 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213467 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213469 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213472 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:46.223145 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213476 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213478 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213481 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213484 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213487 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213490 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213492 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213495 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213498 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213500 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213503 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213505 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213508 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213511 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213513 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213519 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213522 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213525 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213528 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:46.223624 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213531 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213534 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213536 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.213541 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.214829 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.222232 2569 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.222270 2569 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.223033 2569 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.223040 2569 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223117 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223124 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223128 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223131 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223134 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223137 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:46.224081 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223140 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223143 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223146 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223149 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223152 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223154 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223157 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223160 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223163 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223165 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223168 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223171 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223174 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223176 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223179 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223181 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223184 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223187 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223189 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223192 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:46.224467 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223195 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223197 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223200 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223209 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223213 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223216 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223218 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223221 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223224 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223226 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223229 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223232 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223234 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223237 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223240 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223242 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223245 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223247 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223263 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:46.224955 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223267 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223270 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223273 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223276 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223278 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223281 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223284 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223286 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223289 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223291 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223294 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223296 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223299 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223301 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223304 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223314 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223318 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223326 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223329 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223332 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:46.225445 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223335 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223338 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223340 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223343 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223345 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223348 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223352 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223356 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223359 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223362 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223365 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223368 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223371 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223373 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223376 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223379 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223382 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223384 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223387 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223389 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:46.226289 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223392 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223395 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223397 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223400 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223402 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223405 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223408 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223411 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223414 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223419 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223429 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223434 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223437 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223439 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:46.226893 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.223445 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223600 2569 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223605 2569 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223608 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223611 2569 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223615 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223618 2569 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223620 2569 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223623 2569 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223626 2569 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223629 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223631 2569 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223634 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223637 2569 feature_gate.go:328] unrecognized feature gate: Example2 Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223640 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223642 2569 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223645 2569 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223648 2569 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223650 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223653 2569 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 14 18:21:46.227403 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223655 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223658 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223661 2569 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223663 2569 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223666 2569 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223670 2569 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223673 2569 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223677 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223680 2569 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223689 2569 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223692 2569 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223695 2569 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223698 2569 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223701 2569 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223704 2569 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223706 2569 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223709 2569 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223712 2569 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223715 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 14 18:21:46.227884 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223717 2569 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223720 2569 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223722 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223725 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223728 2569 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223730 2569 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223733 2569 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223735 2569 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223738 2569 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223741 2569 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223743 2569 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223746 2569 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223748 2569 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223751 2569 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223753 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223756 2569 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223758 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223761 2569 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223763 2569 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223766 2569 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 14 18:21:46.228357 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223769 2569 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223772 2569 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223775 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223784 2569 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223787 2569 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223790 2569 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223792 2569 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223795 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223797 2569 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223800 2569 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223802 2569 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223806 2569 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223809 2569 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223812 2569 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223815 2569 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223818 2569 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223820 2569 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223823 2569 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223826 2569 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 14 18:21:46.228831 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223830 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223833 2569 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223836 2569 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223839 2569 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223841 2569 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223844 2569 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223846 2569 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223849 2569 feature_gate.go:328] unrecognized feature gate: Example Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223851 2569 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223854 2569 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223856 2569 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223859 2569 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223862 2569 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223865 2569 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223868 2569 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223871 2569 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223873 2569 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223882 2569 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223884 2569 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223887 2569 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 14 18:21:46.229318 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223890 2569 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 14 18:21:46.229804 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:46.223892 2569 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 14 18:21:46.229804 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.223898 2569 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 14 18:21:46.229804 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.224054 2569 server.go:962] "Client rotation is on, will bootstrap in background" Jul 14 18:21:46.229804 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.228375 2569 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 14 18:21:46.230161 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.230148 2569 server.go:1019] "Starting client certificate rotation" Jul 14 18:21:46.230276 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.230245 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:21:46.230308 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.230302 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 14 18:21:46.232459 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.232446 2569 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:21:46.234303 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.234284 2569 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 14 18:21:46.259208 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.259185 2569 log.go:25] "Validated CRI v1 runtime API" Jul 14 18:21:46.261581 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.261553 2569 log.go:25] "Validated CRI v1 image API" Jul 14 18:21:46.263202 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.263175 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:21:46.263409 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.263394 2569 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 14 18:21:46.265945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.265921 2569 fs.go:135] Filesystem UUIDs: map[74be6131-5562-4ced-9d0c-19067c7fae05:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 c254f216-f41b-4254-a04d-96a3d429bc04:/dev/nvme0n1p4] Jul 14 18:21:46.266017 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.265942 2569 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 14 18:21:46.272496 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.272383 2569 manager.go:216] Machine: {Timestamp:2026-07-14 18:21:46.271007892 +0000 UTC m=+0.328675931 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101459 MemoryCapacity:33164488704 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21270def1f9961e1dbab1fbe2cb577 SystemUUID:ec21270d-ef1f-9961-e1db-ab1fbe2cb577 BootID:91210b1e-ed7f-4dc5-9277-e6047abeeea4 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:76:ac:ec:b5:6b Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:76:ac:ec:b5:6b Speed:0 Mtu:9001} {Name:ovs-system MacAddress:da:f6:51:4a:e8:1e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164488704 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 14 18:21:46.272496 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.272492 2569 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 14 18:21:46.272606 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.272586 2569 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 14 18:21:46.275300 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.275267 2569 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 14 18:21:46.275444 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.275302 2569 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 14 18:21:46.275944 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.275933 2569 topology_manager.go:138] "Creating topology manager with none policy" Jul 14 18:21:46.275977 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.275947 2569 container_manager_linux.go:309] "Creating device plugin manager" Jul 14 18:21:46.275977 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.275955 2569 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:21:46.276582 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.276571 2569 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 14 18:21:46.277797 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.277785 2569 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 14 18:21:46.280632 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.280619 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:21:46.280772 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.280762 2569 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 14 18:21:46.280806 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.280793 2569 kubelet.go:486] "Attempting to sync node with API server" Jul 14 18:21:46.280806 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.280803 2569 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 14 18:21:46.280876 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.280818 2569 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 14 18:21:46.280876 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.280830 2569 kubelet.go:398] "Adding apiserver pod source" Jul 14 18:21:46.280876 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.280839 2569 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 14 18:21:46.281236 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.281219 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:21:46.281294 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.281270 2569 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 14 18:21:46.286493 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.286467 2569 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 14 18:21:46.288142 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288127 2569 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 14 18:21:46.288204 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288157 2569 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 14 18:21:46.288440 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288422 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 14 18:21:46.288440 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288441 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 14 18:21:46.288559 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288449 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 14 18:21:46.288559 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288458 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 14 18:21:46.288559 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288470 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 14 18:21:46.288559 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288476 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 14 18:21:46.288896 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288887 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 14 18:21:46.288896 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288896 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 14 18:21:46.288951 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288904 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 14 18:21:46.288951 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288910 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 14 18:21:46.288951 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288918 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 14 18:21:46.288951 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.288926 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 14 18:21:46.289652 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.289638 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 14 18:21:46.289652 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.289649 2569 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 14 18:21:46.292891 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.292874 2569 server.go:1268] "Started kubelet" Jul 14 18:21:46.293062 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.293010 2569 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 14 18:21:46.293159 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.293091 2569 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 14 18:21:46.293159 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.293088 2569 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 14 18:21:46.293335 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.293318 2569 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 14 18:21:46.293390 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.293347 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 14 18:21:46.294170 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.294145 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 14 18:21:46.294281 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.294241 2569 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 14 18:21:46.294236 ip-10-0-130-79 systemd[1]: Started Kubernetes Kubelet. Jul 14 18:21:46.295895 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.295870 2569 server.go:310] "Adding debug handlers to kubelet server" Jul 14 18:21:46.297603 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.297575 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lgvm5" Jul 14 18:21:46.298520 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.298501 2569 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 14 18:21:46.299059 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.298144 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18c23b079df23fdc default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-07-14 18:21:46.292846556 +0000 UTC m=+0.350514596,LastTimestamp:2026-07-14 18:21:46.292846556 +0000 UTC m=+0.350514596,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}" Jul 14 18:21:46.299823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.299806 2569 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 14 18:21:46.300243 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.300227 2569 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 14 18:21:46.300401 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.300385 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:46.300893 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.300871 2569 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 14 18:21:46.300961 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.300900 2569 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 14 18:21:46.300961 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.300878 2569 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 14 18:21:46.301068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301052 2569 reconstruct.go:97] "Volume reconstruction finished" Jul 14 18:21:46.301068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301066 2569 reconciler.go:29] "Reconciler: start to sync state" Jul 14 18:21:46.301213 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301184 2569 factory.go:55] Registering systemd factory Jul 14 18:21:46.301213 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301199 2569 factory.go:223] Registration of the systemd container factory successfully Jul 14 18:21:46.301437 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301426 2569 factory.go:153] Registering CRI-O factory Jul 14 18:21:46.301437 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301438 2569 factory.go:223] Registration of the crio container factory successfully Jul 14 18:21:46.301497 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301478 2569 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 14 18:21:46.301535 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301500 2569 factory.go:103] Registering Raw factory Jul 14 18:21:46.301535 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.301511 2569 manager.go:1196] Started watching for new ooms in manager Jul 14 18:21:46.302187 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.302171 2569 manager.go:349] Starting recovery of all containers Jul 14 18:21:46.309653 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.309628 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lgvm5" Jul 14 18:21:46.311361 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.311333 2569 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 14 18:21:46.311475 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.311454 2569 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 14 18:21:46.313908 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.313697 2569 manager.go:354] Recovery completed Jul 14 18:21:46.318659 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.318646 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:46.321059 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.321042 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:46.321133 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.321073 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:46.321133 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.321087 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:46.321663 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.321650 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:46.321751 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.321651 2569 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 14 18:21:46.321751 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.321707 2569 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 14 18:21:46.321751 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.321724 2569 state_mem.go:36] "Initialized new in-memory state store" Jul 14 18:21:46.322394 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.322376 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:46.322466 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.322411 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:46.322466 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.322452 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:46.322898 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.322881 2569 policy_none.go:49] "None policy: Start" Jul 14 18:21:46.322898 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.322899 2569 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 14 18:21:46.323000 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.322910 2569 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 14 18:21:46.323890 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.323873 2569 policy_none.go:47] "Start" Jul 14 18:21:46.324543 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.323538 2569 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-79.ec2.internal.18c23b079fa0b5b1 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-79.ec2.internal,UID:ip-10-0-130-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-130-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-130-79.ec2.internal,},FirstTimestamp:2026-07-14 18:21:46.321057201 +0000 UTC m=+0.378725240,LastTimestamp:2026-07-14 18:21:46.321057201 +0000 UTC m=+0.378725240,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-79.ec2.internal,}" Jul 14 18:21:46.360919 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.360896 2569 manager.go:337] "Starting Device Plugin manager" Jul 14 18:21:46.361068 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.360934 2569 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 14 18:21:46.361068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.360945 2569 server.go:87] "Starting device plugin registration server" Jul 14 18:21:46.361291 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.361273 2569 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 14 18:21:46.361388 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.361290 2569 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 14 18:21:46.361441 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.361410 2569 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 14 18:21:46.361514 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.361494 2569 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 14 18:21:46.361514 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.361506 2569 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 14 18:21:46.362031 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.362006 2569 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 14 18:21:46.362097 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.362071 2569 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:46.461793 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.461698 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:46.463153 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.462732 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:46.463153 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.462767 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:46.463153 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.462781 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:46.463153 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.462808 2569 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.474661 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.474624 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 14 18:21:46.475890 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.475871 2569 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 14 18:21:46.475976 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.475911 2569 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 14 18:21:46.475976 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.475930 2569 kubelet.go:2448] "Starting kubelet main sync loop" Jul 14 18:21:46.475976 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.475969 2569 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 14 18:21:46.476169 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.476151 2569 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.476269 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.476180 2569 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-79.ec2.internal\": node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:46.479526 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.479509 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:46.576670 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.576630 2569 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal"] Jul 14 18:21:46.576870 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.576714 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:46.577685 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.577665 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:46.577798 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.577702 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:46.577798 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.577717 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:46.579326 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.579314 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:46.579451 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.579426 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.579486 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.579468 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:46.580051 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.580035 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:46.580136 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.580049 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:46.580136 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.580061 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:46.580136 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.580072 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:46.580136 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.580076 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:46.580136 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.580086 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:46.581348 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.581331 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.581400 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.581362 2569 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 14 18:21:46.582103 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.582087 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientMemory" Jul 14 18:21:46.582157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.582118 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasNoDiskPressure" Jul 14 18:21:46.582157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.582128 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeHasSufficientPID" Jul 14 18:21:46.602700 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.602673 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799b04e74e4d564ee24fa3864d962c91-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"799b04e74e4d564ee24fa3864d962c91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.602839 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.602706 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/507da892b8ef03d40dfbcac7df3d206b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"507da892b8ef03d40dfbcac7df3d206b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.602839 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.602730 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/507da892b8ef03d40dfbcac7df3d206b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"507da892b8ef03d40dfbcac7df3d206b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.602839 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.602755 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:46.604237 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.604222 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-79.ec2.internal\" not found" node="ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.608697 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.608680 2569 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-79.ec2.internal\" not found" node="ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.703337 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.703306 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799b04e74e4d564ee24fa3864d962c91-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"799b04e74e4d564ee24fa3864d962c91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.703337 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.703309 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:46.703584 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.703371 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/799b04e74e4d564ee24fa3864d962c91-config\") pod \"kube-apiserver-proxy-ip-10-0-130-79.ec2.internal\" (UID: \"799b04e74e4d564ee24fa3864d962c91\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.703584 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.703421 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/507da892b8ef03d40dfbcac7df3d206b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"507da892b8ef03d40dfbcac7df3d206b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.703584 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.703444 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/507da892b8ef03d40dfbcac7df3d206b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"507da892b8ef03d40dfbcac7df3d206b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.703584 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.703485 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/507da892b8ef03d40dfbcac7df3d206b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"507da892b8ef03d40dfbcac7df3d206b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.703584 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.703458 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/507da892b8ef03d40dfbcac7df3d206b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal\" (UID: \"507da892b8ef03d40dfbcac7df3d206b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.803724 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.803641 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:46.904492 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:46.904457 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:46.906616 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.906597 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:46.911388 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:46.911371 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" Jul 14 18:21:47.005307 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:47.005264 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:47.105692 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:47.105604 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:47.206094 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:47.206054 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:47.229266 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.229233 2569 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 14 18:21:47.229439 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.229422 2569 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 14 18:21:47.307055 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:47.307017 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:47.310120 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.310101 2569 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:47.313691 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.313653 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 18:16:46 +0000 UTC" deadline="2028-04-11 16:54:18.142841498 +0000 UTC" Jul 14 18:21:47.313691 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.313688 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15286h32m30.829158085s" Jul 14 18:21:47.407204 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:47.407174 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:47.489591 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:47.489558 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507da892b8ef03d40dfbcac7df3d206b.slice/crio-8dfef3d7e637ef38919f6cca831adf93a3d6fafb593bead507cb57af538ce71e WatchSource:0}: Error finding container 8dfef3d7e637ef38919f6cca831adf93a3d6fafb593bead507cb57af538ce71e: Status 404 returned error can't find the container with id 8dfef3d7e637ef38919f6cca831adf93a3d6fafb593bead507cb57af538ce71e Jul 14 18:21:47.489799 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:47.489772 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod799b04e74e4d564ee24fa3864d962c91.slice/crio-4ddc70818a80a50b729148426c537a3d5aba173e798cacfc4c727a17acf396b2 WatchSource:0}: Error finding container 4ddc70818a80a50b729148426c537a3d5aba173e798cacfc4c727a17acf396b2: Status 404 returned error can't find the container with id 4ddc70818a80a50b729148426c537a3d5aba173e798cacfc4c727a17acf396b2 Jul 14 18:21:47.495638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.495615 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:21:47.508080 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:47.508048 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:47.555945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.555920 2569 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:47.608125 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:47.608095 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:47.708279 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:47.708223 2569 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-130-79.ec2.internal\" not found" Jul 14 18:21:47.771706 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.771680 2569 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:47.800538 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.800486 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" Jul 14 18:21:47.818284 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.818256 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:21:47.819222 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.819206 2569 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" Jul 14 18:21:47.836498 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:47.836460 2569 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 14 18:21:48.282275 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.282236 2569 apiserver.go:52] "Watching apiserver" Jul 14 18:21:48.295037 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.294991 2569 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 14 18:21:48.296576 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.296485 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-85gbk","openshift-multus/network-metrics-daemon-6dr2s","openshift-network-diagnostics/network-check-target-nlzhr","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x","openshift-cluster-node-tuning-operator/tuned-lcb5d","openshift-image-registry/node-ca-hbw2t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal","openshift-multus/multus-rv8c6","openshift-network-operator/iptables-alerter-nvbbb","openshift-ovn-kubernetes/ovnkube-node-k8sz5","kube-system/global-pull-secret-syncer-rx2bj","kube-system/konnectivity-agent-9rm2m","kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal","openshift-dns/node-resolver-swtrt"] Jul 14 18:21:48.299630 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.298900 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.300392 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.300365 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.300507 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.300452 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:21:48.301295 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.300644 2569 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 14 18:21:48.302191 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.302109 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:48.302291 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.302270 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:21:48.303579 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.303554 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.304450 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.304429 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 14 18:21:48.304668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.304652 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 14 18:21:48.304937 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.304916 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.307721 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.307594 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.308448 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.309378 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gmjgd\"" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.309564 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.309715 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.309811 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.309916 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.310097 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.310123 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.310578 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.310396 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 14 18:21:48.312234 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.311662 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:48.312234 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.311768 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:21:48.312234 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312191 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:21:48.312638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312553 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70bf487c-b158-4aff-9aa0-bc83b031e407-kubelet-config\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.312638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:48.312638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-kubelet\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.312825 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312651 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-ovn\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.312825 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312695 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-log-socket\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.312825 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312742 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-cni-netd\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.312825 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312780 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ljvgt\" (UniqueName: \"kubernetes.io/projected/6f806855-1fa3-49fb-8716-4c934d6886f5-kube-api-access-ljvgt\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.312825 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312817 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70bf487c-b158-4aff-9aa0-bc83b031e407-dbus\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.313109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312847 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-systemd-units\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312872 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-systemd\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312898 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312923 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-node-log\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312947 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-cni-bin\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-ovnkube-script-lib\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.312996 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-slash\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313067 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-run-netns\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313124 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-var-lib-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313161 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-etc-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-ovnkube-config\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313225 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313261 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313310 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313309 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313417 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-env-overrides\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.313678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.313445 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f806855-1fa3-49fb-8716-4c934d6886f5-ovn-node-metrics-cert\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.314186 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314054 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 14 18:21:48.314260 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314224 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-pgp7r\"" Jul 14 18:21:48.314382 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314250 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-98sh2\"" Jul 14 18:21:48.314499 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314386 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 14 18:21:48.314499 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314437 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 14 18:21:48.314607 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314521 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:21:48.314677 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314627 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 14 18:21:48.314677 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314667 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-62f7s\"" Jul 14 18:21:48.314797 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314734 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 14 18:21:48.314865 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314804 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 14 18:21:48.314925 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.314862 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 14 18:21:48.315072 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.315008 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 14 18:21:48.315072 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.315042 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 14 18:21:48.315190 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.315115 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hnlpt\"" Jul 14 18:21:48.315272 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.315235 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.315350 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.315277 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6xhjz\"" Jul 14 18:21:48.316576 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.316550 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:21:48.318363 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.318344 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 14 18:21:48.318806 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.318790 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-66clj\"" Jul 14 18:21:48.319924 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.319903 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 14 18:21:48.320426 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.320406 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 14 18:21:48.320882 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.320862 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 14 18:21:48.321724 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.321707 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 14 18:21:48.322054 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.322035 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 14 18:21:48.322054 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.322049 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 14 18:21:48.322446 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.322428 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-m22vx\"" Jul 14 18:21:48.322639 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.322620 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cr9kq\"" Jul 14 18:21:48.322845 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.322821 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 14 18:21:48.330000 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.329789 2569 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 14 18:21:48.346399 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.346357 2569 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qp7x4" Jul 14 18:21:48.366947 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.366872 2569 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qp7x4" Jul 14 18:21:48.402493 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.402464 2569 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 14 18:21:48.414564 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-ovnkube-config\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.414753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a9e50d0-df10-47d4-9ad8-717bc1af9150-serviceca\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.414753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414607 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-modprobe-d\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.414753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414636 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysconfig\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.414753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414664 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvv9h\" (UniqueName: \"kubernetes.io/projected/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-kube-api-access-wvv9h\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.414753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414690 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.414753 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414716 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63bf62f5-3d91-4db7-879a-91687396744f-cni-binary-copy\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414838 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d4b6cc7-09ea-4477-9224-a59618f6e2d3-agent-certs\") pod \"konnectivity-agent-9rm2m\" (UID: \"2d4b6cc7-09ea-4477-9224-a59618f6e2d3\") " pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414863 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-var-lib-kubelet\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414933 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-socket-dir-parent\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.414964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-log-socket\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415014 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-host\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415041 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-system-cni-dir\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.415076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415056 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-log-socket\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415092 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-system-cni-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-lib-modules\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415158 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-systemd\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415186 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-ovnkube-script-lib\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415215 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415226 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-systemd\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415239 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d4b6cc7-09ea-4477-9224-a59618f6e2d3-konnectivity-ca\") pod \"konnectivity-agent-9rm2m\" (UID: \"2d4b6cc7-09ea-4477-9224-a59618f6e2d3\") " pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415263 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-ovnkube-config\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysctl-d\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415392 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11223d59-522e-4311-ad5c-fcc84cf423e8-tmp-dir\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415434 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-var-lib-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415463 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-env-overrides\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415493 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-var-lib-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415519 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f806855-1fa3-49fb-8716-4c934d6886f5-ovn-node-metrics-cert\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.415549 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415546 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70bf487c-b158-4aff-9aa0-bc83b031e407-kubelet-config\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415572 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415592 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-cni-netd\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415608 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a9e50d0-df10-47d4-9ad8-717bc1af9150-host\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415625 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjcgs\" (UniqueName: \"kubernetes.io/projected/50318283-8577-410a-af69-d1f7a903a1c3-kube-api-access-jjcgs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415657 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/70bf487c-b158-4aff-9aa0-bc83b031e407-kubelet-config\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415661 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-cni-bin\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415699 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-conf-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415707 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-cni-netd\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.415607 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415761 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-node-log\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.415783 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret podName:70bf487c-b158-4aff-9aa0-bc83b031e407 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:48.915767602 +0000 UTC m=+2.973435630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret") pod "global-pull-secret-syncer-rx2bj" (UID: "70bf487c-b158-4aff-9aa0-bc83b031e407") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-node-log\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415810 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-ovnkube-script-lib\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415822 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-socket-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.416157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415838 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-device-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415855 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-kubernetes\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415877 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-cni-binary-copy\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-slash\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415913 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-run-netns\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415929 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-etc-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415945 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2ccl\" (UniqueName: \"kubernetes.io/projected/d37674b0-4e82-4ffc-ac82-05a67ffc413f-kube-api-access-j2ccl\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415961 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11223d59-522e-4311-ad5c-fcc84cf423e8-hosts-file\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415975 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63bf62f5-3d91-4db7-879a-91687396744f-multus-daemon-config\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.415991 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ssvn\" (UniqueName: \"kubernetes.io/projected/11223d59-522e-4311-ad5c-fcc84cf423e8-kube-api-access-6ssvn\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416006 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-cnibin\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416024 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-cni-multus\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416045 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-multus-certs\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416098 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-kubelet\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-ovn\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416150 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ljvgt\" (UniqueName: \"kubernetes.io/projected/6f806855-1fa3-49fb-8716-4c934d6886f5-kube-api-access-ljvgt\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416157 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6f806855-1fa3-49fb-8716-4c934d6886f5-env-overrides\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416175 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70bf487c-b158-4aff-9aa0-bc83b031e407-dbus\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416207 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-host-slash\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416234 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-os-release\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416259 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416300 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-os-release\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416344 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-systemd\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416369 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-sys\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416390 2569 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416448 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-slash\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416489 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-etc-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416623 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/70bf487c-b158-4aff-9aa0-bc83b031e407-dbus\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416700 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-run-netns\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416719 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-kubelet\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416760 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-ovn\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-tuned\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416885 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-tmp\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416932 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.417998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416973 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-run-openvswitch\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.416988 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417024 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysctl-conf\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417051 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-k8s-cni-cncf-io\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417081 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-kubelet\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-hostroot\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417172 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-etc-kubernetes\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417206 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417233 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-run\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417260 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zltjf\" (UniqueName: \"kubernetes.io/projected/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-kube-api-access-zltjf\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417282 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-run-ovn-kubernetes\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417302 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-cnibin\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417327 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46zt4\" (UniqueName: \"kubernetes.io/projected/63bf62f5-3d91-4db7-879a-91687396744f-kube-api-access-46zt4\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417350 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-sys-fs\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417376 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jbd\" (UniqueName: \"kubernetes.io/projected/0a9e50d0-df10-47d4-9ad8-717bc1af9150-kube-api-access-97jbd\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417401 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-iptables-alerter-script\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.418854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417439 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bx7x\" (UniqueName: \"kubernetes.io/projected/93e42db6-7be6-440f-bda1-c9641e64c18e-kube-api-access-5bx7x\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.419794 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417471 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-cni-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.419794 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417495 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-netns\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.419794 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417521 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-systemd-units\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.419794 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417545 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-cni-bin\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.419794 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417575 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-systemd-units\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.419794 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417583 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-registration-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.419794 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417621 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6f806855-1fa3-49fb-8716-4c934d6886f5-host-cni-bin\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.419794 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.417620 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:48.420618 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.420595 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6f806855-1fa3-49fb-8716-4c934d6886f5-ovn-node-metrics-cert\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.425738 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.425708 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:48.425738 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.425736 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:48.425934 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.425752 2569 projected.go:196] Error preparing data for projected volume kube-api-access-6cgmd for pod openshift-network-diagnostics/network-check-target-nlzhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:48.425934 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.425819 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd podName:cb3370dd-1592-4848-b5da-9e27330cd28c nodeName:}" failed. No retries permitted until 2026-07-14 18:21:48.92579825 +0000 UTC m=+2.983466288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-6cgmd" (UniqueName: "kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd") pod "network-check-target-nlzhr" (UID: "cb3370dd-1592-4848-b5da-9e27330cd28c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:48.428824 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.428792 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ljvgt\" (UniqueName: \"kubernetes.io/projected/6f806855-1fa3-49fb-8716-4c934d6886f5-kube-api-access-ljvgt\") pod \"ovnkube-node-k8sz5\" (UID: \"6f806855-1fa3-49fb-8716-4c934d6886f5\") " pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.482205 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.482136 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"507da892b8ef03d40dfbcac7df3d206b","Type":"ContainerStarted","Data":"8dfef3d7e637ef38919f6cca831adf93a3d6fafb593bead507cb57af538ce71e"} Jul 14 18:21:48.483646 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.483614 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" event={"ID":"799b04e74e4d564ee24fa3864d962c91","Type":"ContainerStarted","Data":"4ddc70818a80a50b729148426c537a3d5aba173e798cacfc4c727a17acf396b2"} Jul 14 18:21:48.518945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.518908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-run\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.518945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.518951 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zltjf\" (UniqueName: \"kubernetes.io/projected/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-kube-api-access-zltjf\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.518976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-cnibin\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519000 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46zt4\" (UniqueName: \"kubernetes.io/projected/63bf62f5-3d91-4db7-879a-91687396744f-kube-api-access-46zt4\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519024 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-sys-fs\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519049 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jbd\" (UniqueName: \"kubernetes.io/projected/0a9e50d0-df10-47d4-9ad8-717bc1af9150-kube-api-access-97jbd\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519066 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-run\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519079 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-iptables-alerter-script\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519109 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bx7x\" (UniqueName: \"kubernetes.io/projected/93e42db6-7be6-440f-bda1-c9641e64c18e-kube-api-access-5bx7x\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519131 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-cnibin\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519154 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-cni-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.519200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519198 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-netns\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519225 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-registration-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519250 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519312 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a9e50d0-df10-47d4-9ad8-717bc1af9150-serviceca\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519337 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-modprobe-d\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519360 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysconfig\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519395 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvv9h\" (UniqueName: \"kubernetes.io/projected/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-kube-api-access-wvv9h\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519423 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519450 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63bf62f5-3d91-4db7-879a-91687396744f-cni-binary-copy\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519476 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysconfig\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519490 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-cni-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519610 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-modprobe-d\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.519771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519609 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-sys-fs\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.520399 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d4b6cc7-09ea-4477-9224-a59618f6e2d3-agent-certs\") pod \"konnectivity-agent-9rm2m\" (UID: \"2d4b6cc7-09ea-4477-9224-a59618f6e2d3\") " pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:21:48.520399 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.519988 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/0a9e50d0-df10-47d4-9ad8-717bc1af9150-serviceca\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.520399 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520225 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-iptables-alerter-script\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.520399 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520297 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.520399 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520383 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-netns\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.520657 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.520479 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:48.520657 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.520541 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs podName:50318283-8577-410a-af69-d1f7a903a1c3 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:49.0205234 +0000 UTC m=+3.078191438 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs") pod "network-metrics-daemon-6dr2s" (UID: "50318283-8577-410a-af69-d1f7a903a1c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:48.520657 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-var-lib-kubelet\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.520657 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520595 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-socket-dir-parent\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.520657 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-host\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.520657 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-system-cni-dir\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520675 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-system-cni-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-var-lib-kubelet\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520699 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-lib-modules\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520739 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520770 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d4b6cc7-09ea-4477-9224-a59618f6e2d3-konnectivity-ca\") pod \"konnectivity-agent-9rm2m\" (UID: \"2d4b6cc7-09ea-4477-9224-a59618f6e2d3\") " pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520797 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysctl-d\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520818 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-lib-modules\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520824 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11223d59-522e-4311-ad5c-fcc84cf423e8-tmp-dir\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.520864 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520860 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-etc-selinux\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.520876 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-registration-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521067 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-system-cni-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521113 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-host\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521142 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a9e50d0-df10-47d4-9ad8-717bc1af9150-host\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521166 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjcgs\" (UniqueName: \"kubernetes.io/projected/50318283-8577-410a-af69-d1f7a903a1c3-kube-api-access-jjcgs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521191 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521243 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-cni-bin\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521256 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/11223d59-522e-4311-ad5c-fcc84cf423e8-tmp-dir\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.521306 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-conf-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-socket-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521343 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-device-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521366 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-kubernetes\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521415 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-cni-binary-copy\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521445 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysctl-d\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521448 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2d4b6cc7-09ea-4477-9224-a59618f6e2d3-konnectivity-ca\") pod \"konnectivity-agent-9rm2m\" (UID: \"2d4b6cc7-09ea-4477-9224-a59618f6e2d3\") " pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521502 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-socket-dir-parent\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521445 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2ccl\" (UniqueName: \"kubernetes.io/projected/d37674b0-4e82-4ffc-ac82-05a67ffc413f-kube-api-access-j2ccl\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521653 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-kubernetes\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.521763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521746 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/63bf62f5-3d91-4db7-879a-91687396744f-cni-binary-copy\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521802 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-multus-conf-dir\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521752 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-system-cni-dir\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521804 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11223d59-522e-4311-ad5c-fcc84cf423e8-hosts-file\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521954 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/11223d59-522e-4311-ad5c-fcc84cf423e8-hosts-file\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521996 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63bf62f5-3d91-4db7-879a-91687396744f-multus-daemon-config\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522048 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-device-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.521942 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0a9e50d0-df10-47d4-9ad8-717bc1af9150-host\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522140 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522167 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-socket-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522174 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-cni-bin\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522221 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ssvn\" (UniqueName: \"kubernetes.io/projected/11223d59-522e-4311-ad5c-fcc84cf423e8-kube-api-access-6ssvn\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522222 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/93e42db6-7be6-440f-bda1-c9641e64c18e-cni-binary-copy\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522262 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522270 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-cnibin\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-cni-multus\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522314 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-cnibin\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522408 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-multus-certs\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522417 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-cni-multus\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522453 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-multus-certs\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522469 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-host-slash\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522506 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-os-release\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522535 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-os-release\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522574 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/63bf62f5-3d91-4db7-879a-91687396744f-multus-daemon-config\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522586 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-systemd\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522597 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-os-release\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522609 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-sys\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522536 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-host-slash\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522633 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-tuned\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522652 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/93e42db6-7be6-440f-bda1-c9641e64c18e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522656 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-tmp\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.522945 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522668 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-systemd\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522665 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2d4b6cc7-09ea-4477-9224-a59618f6e2d3-agent-certs\") pod \"konnectivity-agent-9rm2m\" (UID: \"2d4b6cc7-09ea-4477-9224-a59618f6e2d3\") " pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522691 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-os-release\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522701 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-sys\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522725 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522750 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysctl-conf\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-k8s-cni-cncf-io\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-kubelet\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522870 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-hostroot\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522880 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d37674b0-4e82-4ffc-ac82-05a67ffc413f-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-etc-kubernetes\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522912 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-run-k8s-cni-cncf-io\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522977 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-host-var-lib-kubelet\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.522977 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-etc-kubernetes\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.523012 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/63bf62f5-3d91-4db7-879a-91687396744f-hostroot\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.523867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.523029 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-sysctl-conf\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.527428 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.527387 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-tmp\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.527567 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.527498 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-etc-tuned\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.536695 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.536610 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jbd\" (UniqueName: \"kubernetes.io/projected/0a9e50d0-df10-47d4-9ad8-717bc1af9150-kube-api-access-97jbd\") pod \"node-ca-hbw2t\" (UID: \"0a9e50d0-df10-47d4-9ad8-717bc1af9150\") " pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.538821 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.538622 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvv9h\" (UniqueName: \"kubernetes.io/projected/1a36f5c5-145a-4f48-9ad8-f93aeb253b52-kube-api-access-wvv9h\") pod \"iptables-alerter-nvbbb\" (UID: \"1a36f5c5-145a-4f48-9ad8-f93aeb253b52\") " pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.540286 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.540253 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2ccl\" (UniqueName: \"kubernetes.io/projected/d37674b0-4e82-4ffc-ac82-05a67ffc413f-kube-api-access-j2ccl\") pod \"aws-ebs-csi-driver-node-qx74x\" (UID: \"d37674b0-4e82-4ffc-ac82-05a67ffc413f\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.540429 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.540259 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bx7x\" (UniqueName: \"kubernetes.io/projected/93e42db6-7be6-440f-bda1-c9641e64c18e-kube-api-access-5bx7x\") pod \"multus-additional-cni-plugins-85gbk\" (UID: \"93e42db6-7be6-440f-bda1-c9641e64c18e\") " pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.540769 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.540740 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zltjf\" (UniqueName: \"kubernetes.io/projected/9fdd69c0-93f3-4c4a-a87d-06d2853b3d90-kube-api-access-zltjf\") pod \"tuned-lcb5d\" (UID: \"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90\") " pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.541000 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.540975 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjcgs\" (UniqueName: \"kubernetes.io/projected/50318283-8577-410a-af69-d1f7a903a1c3-kube-api-access-jjcgs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:48.541704 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.541660 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46zt4\" (UniqueName: \"kubernetes.io/projected/63bf62f5-3d91-4db7-879a-91687396744f-kube-api-access-46zt4\") pod \"multus-rv8c6\" (UID: \"63bf62f5-3d91-4db7-879a-91687396744f\") " pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.542473 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.542402 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ssvn\" (UniqueName: \"kubernetes.io/projected/11223d59-522e-4311-ad5c-fcc84cf423e8-kube-api-access-6ssvn\") pod \"node-resolver-swtrt\" (UID: \"11223d59-522e-4311-ad5c-fcc84cf423e8\") " pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.618553 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.618517 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:21:48.631013 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.630979 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" Jul 14 18:21:48.643200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.643160 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" Jul 14 18:21:48.654407 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.654374 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-hbw2t" Jul 14 18:21:48.663492 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.663458 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nvbbb" Jul 14 18:21:48.673920 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.673891 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-swtrt" Jul 14 18:21:48.687111 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.687085 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-85gbk" Jul 14 18:21:48.695786 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.695682 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rv8c6" Jul 14 18:21:48.702630 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.702601 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:21:48.834039 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.833940 2569 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 14 18:21:48.925929 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.925888 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:48.925929 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:48.925941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:48.926162 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.926055 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:48.926162 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.926072 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:48.926162 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.926094 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:48.926162 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.926108 2569 projected.go:196] Error preparing data for projected volume kube-api-access-6cgmd for pod openshift-network-diagnostics/network-check-target-nlzhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:48.926162 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.926140 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret podName:70bf487c-b158-4aff-9aa0-bc83b031e407 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:49.92611803 +0000 UTC m=+3.983786083 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret") pod "global-pull-secret-syncer-rx2bj" (UID: "70bf487c-b158-4aff-9aa0-bc83b031e407") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:48.926162 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:48.926159 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd podName:cb3370dd-1592-4848-b5da-9e27330cd28c nodeName:}" failed. No retries permitted until 2026-07-14 18:21:49.926148883 +0000 UTC m=+3.983816917 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cgmd" (UniqueName: "kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd") pod "network-check-target-nlzhr" (UID: "cb3370dd-1592-4848-b5da-9e27330cd28c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:49.026890 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:49.026487 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:49.026890 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.026689 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:49.026890 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.026758 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs podName:50318283-8577-410a-af69-d1f7a903a1c3 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:50.026738487 +0000 UTC m=+4.084406516 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs") pod "network-metrics-daemon-6dr2s" (UID: "50318283-8577-410a-af69-d1f7a903a1c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:49.369885 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:49.369841 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:16:48 +0000 UTC" deadline="2028-03-26 20:12:23.745421708 +0000 UTC" Jul 14 18:21:49.369885 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:49.369878 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14905h50m34.375547833s" Jul 14 18:21:49.476953 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:49.476909 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:49.477142 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.477074 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:21:49.638903 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.638870 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fdd69c0_93f3_4c4a_a87d_06d2853b3d90.slice/crio-89db676d0592114a4d69d521a19be6cda3e33929e4045d3cb24f47d7ce2f1014 WatchSource:0}: Error finding container 89db676d0592114a4d69d521a19be6cda3e33929e4045d3cb24f47d7ce2f1014: Status 404 returned error can't find the container with id 89db676d0592114a4d69d521a19be6cda3e33929e4045d3cb24f47d7ce2f1014 Jul 14 18:21:49.641723 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.641634 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11223d59_522e_4311_ad5c_fcc84cf423e8.slice/crio-9ce72747e1a48a76abf12ed3edb7ff55903bd8768ac0807209540b6dee53eba5 WatchSource:0}: Error finding container 9ce72747e1a48a76abf12ed3edb7ff55903bd8768ac0807209540b6dee53eba5: Status 404 returned error can't find the container with id 9ce72747e1a48a76abf12ed3edb7ff55903bd8768ac0807209540b6dee53eba5 Jul 14 18:21:49.642576 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.642550 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63bf62f5_3d91_4db7_879a_91687396744f.slice/crio-d18fa6706df98b993f8a54784ca7ba1da0437d04cbf8520a7c41caaba49d9c29 WatchSource:0}: Error finding container d18fa6706df98b993f8a54784ca7ba1da0437d04cbf8520a7c41caaba49d9c29: Status 404 returned error can't find the container with id d18fa6706df98b993f8a54784ca7ba1da0437d04cbf8520a7c41caaba49d9c29 Jul 14 18:21:49.646218 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.646155 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f806855_1fa3_49fb_8716_4c934d6886f5.slice/crio-14fb6311f2fb2d3b136e44a3647284b346a74f353288da1e2ea72444805b0d4d WatchSource:0}: Error finding container 14fb6311f2fb2d3b136e44a3647284b346a74f353288da1e2ea72444805b0d4d: Status 404 returned error can't find the container with id 14fb6311f2fb2d3b136e44a3647284b346a74f353288da1e2ea72444805b0d4d Jul 14 18:21:49.646813 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.646671 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93e42db6_7be6_440f_bda1_c9641e64c18e.slice/crio-284c498cde673f730d38dc50d42d06983d4c72fa665cf3753efca3cc0eb9c744 WatchSource:0}: Error finding container 284c498cde673f730d38dc50d42d06983d4c72fa665cf3753efca3cc0eb9c744: Status 404 returned error can't find the container with id 284c498cde673f730d38dc50d42d06983d4c72fa665cf3753efca3cc0eb9c744 Jul 14 18:21:49.648161 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.648136 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9e50d0_df10_47d4_9ad8_717bc1af9150.slice/crio-e42f9f10c1ff6ccd4cbb10db0e24c2c1d8f1838379f393a49aeb4e34e8e1b996 WatchSource:0}: Error finding container e42f9f10c1ff6ccd4cbb10db0e24c2c1d8f1838379f393a49aeb4e34e8e1b996: Status 404 returned error can't find the container with id e42f9f10c1ff6ccd4cbb10db0e24c2c1d8f1838379f393a49aeb4e34e8e1b996 Jul 14 18:21:49.649893 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.649864 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd37674b0_4e82_4ffc_ac82_05a67ffc413f.slice/crio-9be92c65fcf2ebd68b530069e89083c3d90cc0cddbcdf784af7298d414165e65 WatchSource:0}: Error finding container 9be92c65fcf2ebd68b530069e89083c3d90cc0cddbcdf784af7298d414165e65: Status 404 returned error can't find the container with id 9be92c65fcf2ebd68b530069e89083c3d90cc0cddbcdf784af7298d414165e65 Jul 14 18:21:49.650855 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.650757 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a36f5c5_145a_4f48_9ad8_f93aeb253b52.slice/crio-937c182deafab5149b395e877e1923361039c33cd72bd4234952243ea7b0da3f WatchSource:0}: Error finding container 937c182deafab5149b395e877e1923361039c33cd72bd4234952243ea7b0da3f: Status 404 returned error can't find the container with id 937c182deafab5149b395e877e1923361039c33cd72bd4234952243ea7b0da3f Jul 14 18:21:49.652822 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:21:49.652797 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d4b6cc7_09ea_4477_9224_a59618f6e2d3.slice/crio-13aff0ea783cda946e9b3048f2d1ddb6b8488ce6f2237842f25f490067ec7c2d WatchSource:0}: Error finding container 13aff0ea783cda946e9b3048f2d1ddb6b8488ce6f2237842f25f490067ec7c2d: Status 404 returned error can't find the container with id 13aff0ea783cda946e9b3048f2d1ddb6b8488ce6f2237842f25f490067ec7c2d Jul 14 18:21:49.935835 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:49.935562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:49.936049 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:49.936029 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:49.936114 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.935660 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:49.936202 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.936146 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret podName:70bf487c-b158-4aff-9aa0-bc83b031e407 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:51.936124221 +0000 UTC m=+5.993792263 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret") pod "global-pull-secret-syncer-rx2bj" (UID: "70bf487c-b158-4aff-9aa0-bc83b031e407") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:49.936202 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.936197 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:49.936374 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.936217 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:49.936374 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.936230 2569 projected.go:196] Error preparing data for projected volume kube-api-access-6cgmd for pod openshift-network-diagnostics/network-check-target-nlzhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:49.936374 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:49.936300 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd podName:cb3370dd-1592-4848-b5da-9e27330cd28c nodeName:}" failed. No retries permitted until 2026-07-14 18:21:51.936266247 +0000 UTC m=+5.993934275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cgmd" (UniqueName: "kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd") pod "network-check-target-nlzhr" (UID: "cb3370dd-1592-4848-b5da-9e27330cd28c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:50.037792 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.037191 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:50.037792 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:50.037374 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:50.037792 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:50.037443 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs podName:50318283-8577-410a-af69-d1f7a903a1c3 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:52.037424402 +0000 UTC m=+6.095092449 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs") pod "network-metrics-daemon-6dr2s" (UID: "50318283-8577-410a-af69-d1f7a903a1c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:50.370768 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.370638 2569 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 18:16:48 +0000 UTC" deadline="2028-02-24 19:49:48.941143842 +0000 UTC" Jul 14 18:21:50.370768 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.370711 2569 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14161h27m58.570437044s" Jul 14 18:21:50.476621 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.476584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:50.476812 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:50.476730 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:21:50.477296 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.477271 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:50.477420 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:50.477397 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:21:50.489383 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.489243 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nvbbb" event={"ID":"1a36f5c5-145a-4f48-9ad8-f93aeb253b52","Type":"ContainerStarted","Data":"937c182deafab5149b395e877e1923361039c33cd72bd4234952243ea7b0da3f"} Jul 14 18:21:50.490833 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.490790 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" event={"ID":"d37674b0-4e82-4ffc-ac82-05a67ffc413f","Type":"ContainerStarted","Data":"9be92c65fcf2ebd68b530069e89083c3d90cc0cddbcdf784af7298d414165e65"} Jul 14 18:21:50.493139 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.493102 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rv8c6" event={"ID":"63bf62f5-3d91-4db7-879a-91687396744f","Type":"ContainerStarted","Data":"d18fa6706df98b993f8a54784ca7ba1da0437d04cbf8520a7c41caaba49d9c29"} Jul 14 18:21:50.494957 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.494890 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-swtrt" event={"ID":"11223d59-522e-4311-ad5c-fcc84cf423e8","Type":"ContainerStarted","Data":"9ce72747e1a48a76abf12ed3edb7ff55903bd8768ac0807209540b6dee53eba5"} Jul 14 18:21:50.496737 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.496712 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" event={"ID":"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90","Type":"ContainerStarted","Data":"89db676d0592114a4d69d521a19be6cda3e33929e4045d3cb24f47d7ce2f1014"} Jul 14 18:21:50.499268 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.499227 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" event={"ID":"799b04e74e4d564ee24fa3864d962c91","Type":"ContainerStarted","Data":"41b7b8bb4d515d822ad6f5f7fe129fafc3ceb048249a255a67a3cf08e5b95040"} Jul 14 18:21:50.501316 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.501255 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rm2m" event={"ID":"2d4b6cc7-09ea-4477-9224-a59618f6e2d3","Type":"ContainerStarted","Data":"13aff0ea783cda946e9b3048f2d1ddb6b8488ce6f2237842f25f490067ec7c2d"} Jul 14 18:21:50.504863 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.504826 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hbw2t" event={"ID":"0a9e50d0-df10-47d4-9ad8-717bc1af9150","Type":"ContainerStarted","Data":"e42f9f10c1ff6ccd4cbb10db0e24c2c1d8f1838379f393a49aeb4e34e8e1b996"} Jul 14 18:21:50.506603 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.506553 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-85gbk" event={"ID":"93e42db6-7be6-440f-bda1-c9641e64c18e","Type":"ContainerStarted","Data":"284c498cde673f730d38dc50d42d06983d4c72fa665cf3753efca3cc0eb9c744"} Jul 14 18:21:50.508024 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.507984 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"14fb6311f2fb2d3b136e44a3647284b346a74f353288da1e2ea72444805b0d4d"} Jul 14 18:21:50.511250 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.511212 2569 generic.go:356] "Generic (PLEG): container finished" podID="507da892b8ef03d40dfbcac7df3d206b" containerID="25d9ffd8630511885356cdbeb922d43bff83f7325e58ca103d42262e188d381d" exitCode=0 Jul 14 18:21:50.511250 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.511255 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"507da892b8ef03d40dfbcac7df3d206b","Type":"ContainerDied","Data":"25d9ffd8630511885356cdbeb922d43bff83f7325e58ca103d42262e188d381d"} Jul 14 18:21:50.567816 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:50.567744 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-79.ec2.internal" podStartSLOduration=2.567716655 podStartE2EDuration="2.567716655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:21:50.537868778 +0000 UTC m=+4.595536822" watchObservedRunningTime="2026-07-14 18:21:50.567716655 +0000 UTC m=+4.625384708" Jul 14 18:21:51.476934 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:51.476846 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:51.477498 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:51.477006 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:21:51.518643 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:51.518393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" event={"ID":"507da892b8ef03d40dfbcac7df3d206b","Type":"ContainerStarted","Data":"ea5a7148b954680291cb1a036f1be113cd626f43ad32397cd8f6cc543d5192c7"} Jul 14 18:21:51.954587 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:51.954474 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:51.954587 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:51.954537 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:51.954828 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:51.954716 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:51.954828 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:51.954741 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:51.954828 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:51.954754 2569 projected.go:196] Error preparing data for projected volume kube-api-access-6cgmd for pod openshift-network-diagnostics/network-check-target-nlzhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:51.954828 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:51.954818 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd podName:cb3370dd-1592-4848-b5da-9e27330cd28c nodeName:}" failed. No retries permitted until 2026-07-14 18:21:55.954798407 +0000 UTC m=+10.012466456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cgmd" (UniqueName: "kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd") pod "network-check-target-nlzhr" (UID: "cb3370dd-1592-4848-b5da-9e27330cd28c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:51.955197 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:51.955170 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:51.955284 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:51.955253 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret podName:70bf487c-b158-4aff-9aa0-bc83b031e407 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:55.955233242 +0000 UTC m=+10.012901278 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret") pod "global-pull-secret-syncer-rx2bj" (UID: "70bf487c-b158-4aff-9aa0-bc83b031e407") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:52.055907 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:52.055867 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:52.056107 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:52.056057 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:52.056179 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:52.056144 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs podName:50318283-8577-410a-af69-d1f7a903a1c3 nodeName:}" failed. No retries permitted until 2026-07-14 18:21:56.056121519 +0000 UTC m=+10.113789551 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs") pod "network-metrics-daemon-6dr2s" (UID: "50318283-8577-410a-af69-d1f7a903a1c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:52.477081 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:52.477049 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:52.477554 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:52.477190 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:21:52.477554 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:52.477210 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:52.477554 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:52.477368 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:21:53.477234 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:53.477169 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:53.477726 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:53.477352 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:21:54.476707 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:54.476622 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:54.476878 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:54.476758 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:21:54.477325 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:54.477283 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:54.477683 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:54.477414 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:21:55.476841 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:55.476673 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:55.476841 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:55.476807 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:21:55.992666 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:55.992624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:55.993148 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:55.992677 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:55.993148 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:55.992822 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:55.993148 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:55.992871 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:21:55.993148 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:55.992888 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:21:55.993148 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:55.992897 2569 projected.go:196] Error preparing data for projected volume kube-api-access-6cgmd for pod openshift-network-diagnostics/network-check-target-nlzhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:55.993148 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:55.992901 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret podName:70bf487c-b158-4aff-9aa0-bc83b031e407 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:03.992878854 +0000 UTC m=+18.050546898 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret") pod "global-pull-secret-syncer-rx2bj" (UID: "70bf487c-b158-4aff-9aa0-bc83b031e407") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:21:55.993148 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:55.992962 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd podName:cb3370dd-1592-4848-b5da-9e27330cd28c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:03.992928382 +0000 UTC m=+18.050596424 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cgmd" (UniqueName: "kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd") pod "network-check-target-nlzhr" (UID: "cb3370dd-1592-4848-b5da-9e27330cd28c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:21:56.093565 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:56.093532 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:56.093765 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:56.093660 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:56.093765 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:56.093726 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs podName:50318283-8577-410a-af69-d1f7a903a1c3 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:04.093708174 +0000 UTC m=+18.151376209 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs") pod "network-metrics-daemon-6dr2s" (UID: "50318283-8577-410a-af69-d1f7a903a1c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:21:56.478050 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:56.477388 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:56.478050 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:56.477425 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:56.478050 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:56.477511 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:21:56.478050 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:56.477954 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:21:57.476592 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:57.476497 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:57.477012 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:57.476649 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:21:57.533105 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:57.533045 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9rm2m" event={"ID":"2d4b6cc7-09ea-4477-9224-a59618f6e2d3","Type":"ContainerStarted","Data":"931136059c9f7afd075ac272f43d0b4eea8358cd432ab1bc6ca109b66b19293b"} Jul 14 18:21:57.550892 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:57.550818 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-79.ec2.internal" podStartSLOduration=9.550791281 podStartE2EDuration="9.550791281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:21:51.562075563 +0000 UTC m=+5.619743615" watchObservedRunningTime="2026-07-14 18:21:57.550791281 +0000 UTC m=+11.608459330" Jul 14 18:21:57.551339 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:57.551303 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9rm2m" podStartSLOduration=3.168528645 podStartE2EDuration="10.551290887s" totalImagesPullingTime="7.382762242s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:21:57.550184625 +0000 UTC m=+11.607852698" watchObservedRunningTime="2026-07-14 18:21:57.551290887 +0000 UTC m=+11.608958936" Jul 14 18:21:58.477433 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:58.477219 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:21:58.477433 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:58.477393 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:21:58.477958 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:58.477489 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:21:58.477958 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:58.477605 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:21:59.476788 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:21:59.476750 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:21:59.476968 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:21:59.476891 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:00.477081 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:00.477049 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:00.477600 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:00.477061 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:00.477600 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:00.477161 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:00.477600 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:00.477278 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:01.476817 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:01.476773 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:01.477021 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:01.476934 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:02.027900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:02.027864 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:22:02.028771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:02.028746 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:22:02.477238 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:02.477206 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:02.477437 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:02.477206 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:02.477437 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:02.477352 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:02.477542 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:02.477451 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:03.477043 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.477012 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:03.477649 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:03.477160 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:03.547709 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.547626 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-hbw2t" event={"ID":"0a9e50d0-df10-47d4-9ad8-717bc1af9150","Type":"ContainerStarted","Data":"5425c018e5f19594bdceb72d23103c00acc06175ca37a8d6c5d656298a57f07a"} Jul 14 18:22:03.549494 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.549457 2569 generic.go:356] "Generic (PLEG): container finished" podID="93e42db6-7be6-440f-bda1-c9641e64c18e" containerID="f7f889830713b840c014b5823d27fa2963f9c6be531a29e6ed335aefebd1ae66" exitCode=0 Jul 14 18:22:03.549675 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.549550 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-85gbk" event={"ID":"93e42db6-7be6-440f-bda1-c9641e64c18e","Type":"ContainerDied","Data":"f7f889830713b840c014b5823d27fa2963f9c6be531a29e6ed335aefebd1ae66"} Jul 14 18:22:03.552451 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.552421 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" event={"ID":"d37674b0-4e82-4ffc-ac82-05a67ffc413f","Type":"ContainerStarted","Data":"e5a0440a61925929681860cef4d111a3d42d184a61ce82b57852c9bc9af479f5"} Jul 14 18:22:03.554843 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.554811 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rv8c6" event={"ID":"63bf62f5-3d91-4db7-879a-91687396744f","Type":"ContainerStarted","Data":"68104c98d1745ba8867be4b5afa987dbf7b305c5e84260c25155968a51ab14c2"} Jul 14 18:22:03.556891 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.556658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-swtrt" event={"ID":"11223d59-522e-4311-ad5c-fcc84cf423e8","Type":"ContainerStarted","Data":"902dcef8762bdb9374c24a4216d27e9610c0af6f73ca115b57393013c10a2af4"} Jul 14 18:22:03.558453 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.558421 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" event={"ID":"9fdd69c0-93f3-4c4a-a87d-06d2853b3d90","Type":"ContainerStarted","Data":"1cc9e8bcad563ecd02b1d61046bf28b721e80a508e936cbe9cbd575c727a0c50"} Jul 14 18:22:03.569874 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.569803 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-hbw2t" podStartSLOduration=9.172613026 podStartE2EDuration="16.569782022s" totalImagesPullingTime="7.397168996s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:03.569764494 +0000 UTC m=+17.627432571" watchObservedRunningTime="2026-07-14 18:22:03.569782022 +0000 UTC m=+17.627450072" Jul 14 18:22:03.601275 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.601209 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rv8c6" podStartSLOduration=3.614773267 podStartE2EDuration="16.601192851s" totalImagesPullingTime="12.986419584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:03.600663132 +0000 UTC m=+17.658331182" watchObservedRunningTime="2026-07-14 18:22:03.601192851 +0000 UTC m=+17.658860899" Jul 14 18:22:03.626406 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.626348 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lcb5d" podStartSLOduration=3.691849246 podStartE2EDuration="16.626325718s" totalImagesPullingTime="12.934476472s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:03.62629131 +0000 UTC m=+17.683959375" watchObservedRunningTime="2026-07-14 18:22:03.626325718 +0000 UTC m=+17.683993765" Jul 14 18:22:03.643733 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.643681 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-swtrt" podStartSLOduration=3.710184828 podStartE2EDuration="16.643664069s" totalImagesPullingTime="12.933479241s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:03.643361061 +0000 UTC m=+17.701029149" watchObservedRunningTime="2026-07-14 18:22:03.643664069 +0000 UTC m=+17.701332117" Jul 14 18:22:03.657104 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.657069 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:22:03.657291 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.657240 2569 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 14 18:22:03.657965 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:03.657940 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9rm2m" Jul 14 18:22:04.059332 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:04.059284 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:04.059520 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:04.059344 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:04.059520 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.059477 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:04.059671 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.059567 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret podName:70bf487c-b158-4aff-9aa0-bc83b031e407 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:20.059544431 +0000 UTC m=+34.117212471 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret") pod "global-pull-secret-syncer-rx2bj" (UID: "70bf487c-b158-4aff-9aa0-bc83b031e407") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:04.059671 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.059486 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:22:04.059671 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.059593 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:22:04.059671 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.059611 2569 projected.go:196] Error preparing data for projected volume kube-api-access-6cgmd for pod openshift-network-diagnostics/network-check-target-nlzhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:04.059671 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.059656 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd podName:cb3370dd-1592-4848-b5da-9e27330cd28c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:20.059644183 +0000 UTC m=+34.117312209 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cgmd" (UniqueName: "kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd") pod "network-check-target-nlzhr" (UID: "cb3370dd-1592-4848-b5da-9e27330cd28c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:04.160456 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:04.160425 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:04.160647 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.160605 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:04.160706 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.160694 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs podName:50318283-8577-410a-af69-d1f7a903a1c3 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:20.160670827 +0000 UTC m=+34.218338872 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs") pod "network-metrics-daemon-6dr2s" (UID: "50318283-8577-410a-af69-d1f7a903a1c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:04.477221 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:04.477124 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:04.477875 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.477280 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:04.477875 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:04.477315 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:04.477875 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:04.477447 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:04.562773 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:04.562738 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nvbbb" event={"ID":"1a36f5c5-145a-4f48-9ad8-f93aeb253b52","Type":"ContainerStarted","Data":"07448b7ce52dfab39796a3eb199ba218633cea4141160f259f29f4397a992105"} Jul 14 18:22:04.598007 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:04.597942 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nvbbb" podStartSLOduration=10.207700335 podStartE2EDuration="17.597915535s" totalImagesPullingTime="7.3902152s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:04.597817387 +0000 UTC m=+18.655485464" watchObservedRunningTime="2026-07-14 18:22:04.597915535 +0000 UTC m=+18.655583585" Jul 14 18:22:05.476145 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:05.476109 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:05.476335 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:05.476234 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:06.477811 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:06.477776 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:06.478497 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:06.477888 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:06.478497 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:06.477946 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:06.478497 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:06.478085 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:07.477212 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:07.477012 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:07.477399 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:07.477316 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:08.476619 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:08.476584 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:08.477274 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:08.476594 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:08.477274 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:08.476726 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:08.477274 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:08.476881 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:09.476755 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:09.476714 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:09.477235 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:09.476867 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:10.476563 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:10.476521 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:10.476745 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:10.476525 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:10.476745 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:10.476662 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:10.477098 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:10.476749 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:11.018910 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.018879 2569 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 14 18:22:11.376299 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.376039 2569 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T18:22:11.018901727Z","UUID":"365f0762-c3bf-41bb-b67f-f4868ff86255","Handler":null,"Name":"","Endpoint":""} Jul 14 18:22:11.377699 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.377682 2569 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 14 18:22:11.377783 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.377717 2569 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 14 18:22:11.477118 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.477097 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:11.477548 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:11.477193 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:11.581344 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.581308 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" event={"ID":"d37674b0-4e82-4ffc-ac82-05a67ffc413f","Type":"ContainerStarted","Data":"a93482a92b1dee403cbd5e1a8e2e9b24ef0c86a80be45c9b3dfb5767938e3a81"} Jul 14 18:22:11.583596 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.583562 2569 generic.go:356] "Generic (PLEG): container finished" podID="93e42db6-7be6-440f-bda1-c9641e64c18e" containerID="5f49471ffc1bea446033087bbdf141938eb7255b5460bc6e6669b6a7f0af547c" exitCode=0 Jul 14 18:22:11.583748 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.583635 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-85gbk" event={"ID":"93e42db6-7be6-440f-bda1-c9641e64c18e","Type":"ContainerDied","Data":"5f49471ffc1bea446033087bbdf141938eb7255b5460bc6e6669b6a7f0af547c"} Jul 14 18:22:11.587545 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.587518 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"c4c214d77f8c085cb16fc52c8a6b91b72581eb13ec748041e9edbb705a024564"} Jul 14 18:22:11.587617 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.587610 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"6dbdb92036c6c4f3f061708570e047d274180d1d6f95f47743bdff96a919bd81"} Jul 14 18:22:11.587650 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.587641 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"5d991c573fbe27df9847a363cfac214d9ddfba697a1f3ee0990f1d6fe4bfe045"} Jul 14 18:22:11.587692 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.587669 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"12f94b75a6c0b6cb1aeed0a6312578f00be9326c0d3953235f8034046c51ab28"} Jul 14 18:22:11.587728 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.587697 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"d14652e084eaf7369de92b058e102a0bee1fe4f7e742c2858a3ad74aed405e14"} Jul 14 18:22:11.587728 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:11.587725 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"63f8b046b10f9f233604496b2ab85187fc042c94571b7b943930ee16fee77206"} Jul 14 18:22:12.476416 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:12.476385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:12.476582 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:12.476503 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:12.476582 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:12.476561 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:12.476686 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:12.476666 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:12.591604 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:12.591516 2569 generic.go:356] "Generic (PLEG): container finished" podID="93e42db6-7be6-440f-bda1-c9641e64c18e" containerID="0f6d2b8c412a9baf952db7dae67c2cf35070514cf75c9d098cc852bdb8262f5b" exitCode=0 Jul 14 18:22:12.591997 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:12.591600 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-85gbk" event={"ID":"93e42db6-7be6-440f-bda1-c9641e64c18e","Type":"ContainerDied","Data":"0f6d2b8c412a9baf952db7dae67c2cf35070514cf75c9d098cc852bdb8262f5b"} Jul 14 18:22:12.593724 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:12.593702 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" event={"ID":"d37674b0-4e82-4ffc-ac82-05a67ffc413f","Type":"ContainerStarted","Data":"f6eb4800af5bf0c6c13d9024347c3723742272c2720afba2960f97bd9af064c7"} Jul 14 18:22:12.653274 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:12.653214 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qx74x" podStartSLOduration=3.410863475 podStartE2EDuration="25.653199706s" totalImagesPullingTime="22.242336231s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:12.652889817 +0000 UTC m=+26.710557866" watchObservedRunningTime="2026-07-14 18:22:12.653199706 +0000 UTC m=+26.710867755" Jul 14 18:22:13.476230 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:13.476139 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:13.476386 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:13.476269 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:13.598111 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:13.598082 2569 generic.go:356] "Generic (PLEG): container finished" podID="93e42db6-7be6-440f-bda1-c9641e64c18e" containerID="1d5b339dcd24cdd75053b87adeee1978bb6a85fd8d8e8bfae98854799fbbd8e6" exitCode=0 Jul 14 18:22:13.598715 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:13.598171 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-85gbk" event={"ID":"93e42db6-7be6-440f-bda1-c9641e64c18e","Type":"ContainerDied","Data":"1d5b339dcd24cdd75053b87adeee1978bb6a85fd8d8e8bfae98854799fbbd8e6"} Jul 14 18:22:14.479481 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:14.479404 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:14.479616 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:14.479501 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:14.479616 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:14.479548 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:14.479690 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:14.479635 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:14.603780 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:14.603739 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"88f5a1754450ae2f958cf04d3d9303b12ca8c9ad2a8418dcc98f1336f5e6c43e"} Jul 14 18:22:15.476798 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:15.476762 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:15.476963 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:15.476870 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:16.480782 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:16.480530 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:16.481298 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:16.480565 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:16.481298 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:16.480895 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:16.481298 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:16.480916 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:16.613391 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:16.613347 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" event={"ID":"6f806855-1fa3-49fb-8716-4c934d6886f5","Type":"ContainerStarted","Data":"27787c1383e676a19f2e37275f2d1bc7dfb143a81d61ed945d874b866f9f370f"} Jul 14 18:22:16.613683 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:16.613649 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:22:16.632573 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:16.632547 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:22:16.691810 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:16.691752 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" podStartSLOduration=8.433145801 podStartE2EDuration="29.691736349s" totalImagesPullingTime="21.258590548s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:16.652769931 +0000 UTC m=+30.710437999" watchObservedRunningTime="2026-07-14 18:22:16.691736349 +0000 UTC m=+30.749404398" Jul 14 18:22:17.476491 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:17.476458 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:17.476656 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:17.476576 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:17.616500 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:17.616456 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:22:17.616951 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:17.616552 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:22:17.634493 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:17.634463 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:22:18.216064 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:18.216014 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rx2bj"] Jul 14 18:22:18.216294 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:18.216132 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:18.216294 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:18.216245 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:18.219906 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:18.219876 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nlzhr"] Jul 14 18:22:18.220039 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:18.220000 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:18.220148 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:18.220110 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:18.221037 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:18.221011 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6dr2s"] Jul 14 18:22:18.221147 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:18.221128 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:18.221247 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:18.221224 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:19.476624 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:19.476536 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:19.476985 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:19.476645 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:19.622733 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:19.622700 2569 generic.go:356] "Generic (PLEG): container finished" podID="93e42db6-7be6-440f-bda1-c9641e64c18e" containerID="28c2d07e9659fa60cf89250779752ebe91ae9a854a498ed68bbfba900a697fc1" exitCode=0 Jul 14 18:22:19.622910 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:19.622792 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-85gbk" event={"ID":"93e42db6-7be6-440f-bda1-c9641e64c18e","Type":"ContainerDied","Data":"28c2d07e9659fa60cf89250779752ebe91ae9a854a498ed68bbfba900a697fc1"} Jul 14 18:22:20.077818 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:20.077728 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:20.077818 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:20.077766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:20.078194 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.077873 2569 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:20.078194 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.077888 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 14 18:22:20.078194 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.077901 2569 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 14 18:22:20.078194 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.077915 2569 projected.go:196] Error preparing data for projected volume kube-api-access-6cgmd for pod openshift-network-diagnostics/network-check-target-nlzhr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:20.078194 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.077940 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret podName:70bf487c-b158-4aff-9aa0-bc83b031e407 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:52.077922005 +0000 UTC m=+66.135590055 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret") pod "global-pull-secret-syncer-rx2bj" (UID: "70bf487c-b158-4aff-9aa0-bc83b031e407") : object "kube-system"/"original-pull-secret" not registered Jul 14 18:22:20.078194 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.077957 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd podName:cb3370dd-1592-4848-b5da-9e27330cd28c nodeName:}" failed. No retries permitted until 2026-07-14 18:22:52.077949943 +0000 UTC m=+66.135617974 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-6cgmd" (UniqueName: "kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd") pod "network-check-target-nlzhr" (UID: "cb3370dd-1592-4848-b5da-9e27330cd28c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 14 18:22:20.178736 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:20.178703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:20.178917 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.178841 2569 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:20.178917 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.178910 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs podName:50318283-8577-410a-af69-d1f7a903a1c3 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:52.178894007 +0000 UTC m=+66.236562049 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs") pod "network-metrics-daemon-6dr2s" (UID: "50318283-8577-410a-af69-d1f7a903a1c3") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 14 18:22:20.479157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:20.479073 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:20.479157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:20.479115 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:20.479665 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.479190 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:20.479665 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:20.479307 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:20.627392 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:20.627363 2569 generic.go:356] "Generic (PLEG): container finished" podID="93e42db6-7be6-440f-bda1-c9641e64c18e" containerID="dcbec1e16d509fd8f5853e9959f61dda97a4c2175328646ec5efd5d0d4843ffd" exitCode=0 Jul 14 18:22:20.627560 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:20.627414 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-85gbk" event={"ID":"93e42db6-7be6-440f-bda1-c9641e64c18e","Type":"ContainerDied","Data":"dcbec1e16d509fd8f5853e9959f61dda97a4c2175328646ec5efd5d0d4843ffd"} Jul 14 18:22:21.476605 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:21.476573 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:21.476777 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:21.476688 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dr2s" podUID="50318283-8577-410a-af69-d1f7a903a1c3" Jul 14 18:22:21.633447 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:21.633367 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-85gbk" event={"ID":"93e42db6-7be6-440f-bda1-c9641e64c18e","Type":"ContainerStarted","Data":"b5c7b1ea24b345401a97afafde391db2456e46dfb1fd6b0d5cf05078b6a16294"} Jul 14 18:22:21.679164 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:21.679101 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-85gbk" podStartSLOduration=12.950872884 podStartE2EDuration="34.679084479s" totalImagesPullingTime="21.728211595s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:21.675655946 +0000 UTC m=+35.733323994" watchObservedRunningTime="2026-07-14 18:22:21.679084479 +0000 UTC m=+35.736752528" Jul 14 18:22:22.481941 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:22.481913 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:22.482109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:22.481909 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:22.482109 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:22.482019 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nlzhr" podUID="cb3370dd-1592-4848-b5da-9e27330cd28c" Jul 14 18:22:22.482109 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:22.482072 2569 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-rx2bj" podUID="70bf487c-b158-4aff-9aa0-bc83b031e407" Jul 14 18:22:23.244843 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.244816 2569 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-130-79.ec2.internal" event="NodeReady" Jul 14 18:22:23.245194 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.244941 2569 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 14 18:22:23.296149 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.296114 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-585dd8c8cc-f2d22"] Jul 14 18:22:23.315666 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.315630 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.317274 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.317229 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-z5d6b"] Jul 14 18:22:23.319872 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.319853 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 14 18:22:23.320389 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.320374 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-psptv\"" Jul 14 18:22:23.320890 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.320844 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 14 18:22:23.320890 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.320844 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 14 18:22:23.335854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.335775 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hnl7l"] Jul 14 18:22:23.335854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.335803 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z5d6b" Jul 14 18:22:23.336076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.335930 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 14 18:22:23.338800 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.338595 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 14 18:22:23.339413 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.339388 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 14 18:22:23.339413 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.339406 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 14 18:22:23.339983 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.339967 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-58lnk\"" Jul 14 18:22:23.359869 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.359838 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585dd8c8cc-f2d22"] Jul 14 18:22:23.359869 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.359875 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z5d6b"] Jul 14 18:22:23.360068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.359891 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hnl7l"] Jul 14 18:22:23.360068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.359951 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.363027 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.363002 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 14 18:22:23.363173 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.363155 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wk6vb\"" Jul 14 18:22:23.363317 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.363303 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 14 18:22:23.399003 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.398971 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.399211 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.399019 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96f143d8-874e-4d12-94c5-51227b2d0740-ca-trust-extracted\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.399211 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.399066 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-image-registry-private-configuration\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.399211 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.399088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-trusted-ca\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.399211 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.399119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-kube-api-access-bkd4r\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.399211 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.399196 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-registry-certificates\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.399457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.399233 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-installation-pull-secrets\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.399457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.399289 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-bound-sa-token\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.476957 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.476922 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:23.480002 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.479980 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:22:23.480148 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.480116 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzvpc\"" Jul 14 18:22:23.499651 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.499614 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-image-registry-private-configuration\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.499651 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.499652 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-trusted-ca\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.499910 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.499674 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghhwm\" (UniqueName: \"kubernetes.io/projected/5a06092a-5be1-49bf-98d3-4f47ae464faf-kube-api-access-ghhwm\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.499910 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.499790 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxgft\" (UniqueName: \"kubernetes.io/projected/0cf86795-02bf-4e52-a5ac-ae404d64f9bd-kube-api-access-mxgft\") pod \"ingress-canary-z5d6b\" (UID: \"0cf86795-02bf-4e52-a5ac-ae404d64f9bd\") " pod="openshift-ingress-canary/ingress-canary-z5d6b" Jul 14 18:22:23.499910 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.499828 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf86795-02bf-4e52-a5ac-ae404d64f9bd-cert\") pod \"ingress-canary-z5d6b\" (UID: \"0cf86795-02bf-4e52-a5ac-ae404d64f9bd\") " pod="openshift-ingress-canary/ingress-canary-z5d6b" Jul 14 18:22:23.499910 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.499855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-kube-api-access-bkd4r\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.499910 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.499891 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a06092a-5be1-49bf-98d3-4f47ae464faf-tmp-dir\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.500133 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500019 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-registry-certificates\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.500133 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500055 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-installation-pull-secrets\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.500133 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-bound-sa-token\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.500133 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500120 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a06092a-5be1-49bf-98d3-4f47ae464faf-config-volume\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.500324 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500142 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.500324 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500174 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.500324 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500232 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96f143d8-874e-4d12-94c5-51227b2d0740-ca-trust-extracted\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.500460 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:23.500339 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:23.500460 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:23.500356 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585dd8c8cc-f2d22: secret "image-registry-tls" not found Jul 14 18:22:23.500460 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:23.500428 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls podName:96f143d8-874e-4d12-94c5-51227b2d0740 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:24.00040863 +0000 UTC m=+38.058076667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls") pod "image-registry-585dd8c8cc-f2d22" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740") : secret "image-registry-tls" not found Jul 14 18:22:23.500625 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500589 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96f143d8-874e-4d12-94c5-51227b2d0740-ca-trust-extracted\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.500677 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500637 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-registry-certificates\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.500677 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.500660 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-trusted-ca\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.503967 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.503941 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-installation-pull-secrets\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.504202 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.504043 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-image-registry-private-configuration\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.523760 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.523717 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-bound-sa-token\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.524576 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.524554 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-kube-api-access-bkd4r\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:23.601244 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.601153 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghhwm\" (UniqueName: \"kubernetes.io/projected/5a06092a-5be1-49bf-98d3-4f47ae464faf-kube-api-access-ghhwm\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.601244 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.601200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxgft\" (UniqueName: \"kubernetes.io/projected/0cf86795-02bf-4e52-a5ac-ae404d64f9bd-kube-api-access-mxgft\") pod \"ingress-canary-z5d6b\" (UID: \"0cf86795-02bf-4e52-a5ac-ae404d64f9bd\") " pod="openshift-ingress-canary/ingress-canary-z5d6b" Jul 14 18:22:23.601244 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.601222 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf86795-02bf-4e52-a5ac-ae404d64f9bd-cert\") pod \"ingress-canary-z5d6b\" (UID: \"0cf86795-02bf-4e52-a5ac-ae404d64f9bd\") " pod="openshift-ingress-canary/ingress-canary-z5d6b" Jul 14 18:22:23.601557 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.601278 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a06092a-5be1-49bf-98d3-4f47ae464faf-tmp-dir\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.601557 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.601307 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a06092a-5be1-49bf-98d3-4f47ae464faf-config-volume\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.601557 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.601323 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.601557 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:23.601415 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:23.601557 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:23.601496 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls podName:5a06092a-5be1-49bf-98d3-4f47ae464faf nodeName:}" failed. No retries permitted until 2026-07-14 18:22:24.101477193 +0000 UTC m=+38.159145221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls") pod "dns-default-hnl7l" (UID: "5a06092a-5be1-49bf-98d3-4f47ae464faf") : secret "dns-default-metrics-tls" not found Jul 14 18:22:23.601821 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.601734 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5a06092a-5be1-49bf-98d3-4f47ae464faf-tmp-dir\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.601970 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.601949 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/5a06092a-5be1-49bf-98d3-4f47ae464faf-config-volume\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.603744 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.603694 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0cf86795-02bf-4e52-a5ac-ae404d64f9bd-cert\") pod \"ingress-canary-z5d6b\" (UID: \"0cf86795-02bf-4e52-a5ac-ae404d64f9bd\") " pod="openshift-ingress-canary/ingress-canary-z5d6b" Jul 14 18:22:23.611093 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.611064 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxgft\" (UniqueName: \"kubernetes.io/projected/0cf86795-02bf-4e52-a5ac-ae404d64f9bd-kube-api-access-mxgft\") pod \"ingress-canary-z5d6b\" (UID: \"0cf86795-02bf-4e52-a5ac-ae404d64f9bd\") " pod="openshift-ingress-canary/ingress-canary-z5d6b" Jul 14 18:22:23.611208 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.611066 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghhwm\" (UniqueName: \"kubernetes.io/projected/5a06092a-5be1-49bf-98d3-4f47ae464faf-kube-api-access-ghhwm\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:23.645118 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.645086 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-z5d6b" Jul 14 18:22:23.850461 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:23.850285 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-z5d6b"] Jul 14 18:22:23.853182 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:23.853115 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf86795_02bf_4e52_a5ac_ae404d64f9bd.slice/crio-fe1f368f157703bc5e55df26298b1e5589646346848f74b4a194e9bc9b481032 WatchSource:0}: Error finding container fe1f368f157703bc5e55df26298b1e5589646346848f74b4a194e9bc9b481032: Status 404 returned error can't find the container with id fe1f368f157703bc5e55df26298b1e5589646346848f74b4a194e9bc9b481032 Jul 14 18:22:24.004224 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.004179 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:24.004408 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:24.004248 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:24.004408 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:24.004280 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585dd8c8cc-f2d22: secret "image-registry-tls" not found Jul 14 18:22:24.004408 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:24.004333 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls podName:96f143d8-874e-4d12-94c5-51227b2d0740 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:25.004319507 +0000 UTC m=+39.061987551 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls") pod "image-registry-585dd8c8cc-f2d22" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740") : secret "image-registry-tls" not found Jul 14 18:22:24.105302 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.105189 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:24.105456 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:24.105361 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:24.105456 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:24.105422 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls podName:5a06092a-5be1-49bf-98d3-4f47ae464faf nodeName:}" failed. No retries permitted until 2026-07-14 18:22:25.105407314 +0000 UTC m=+39.163075342 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls") pod "dns-default-hnl7l" (UID: "5a06092a-5be1-49bf-98d3-4f47ae464faf") : secret "dns-default-metrics-tls" not found Jul 14 18:22:24.476850 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.476770 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:24.477441 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.476908 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:24.480589 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.480552 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:22:24.481890 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.481853 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:22:24.482019 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.481888 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:22:24.482019 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.481889 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lzfqw\"" Jul 14 18:22:24.642375 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:24.642332 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z5d6b" event={"ID":"0cf86795-02bf-4e52-a5ac-ae404d64f9bd","Type":"ContainerStarted","Data":"fe1f368f157703bc5e55df26298b1e5589646346848f74b4a194e9bc9b481032"} Jul 14 18:22:25.018855 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:25.018819 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:25.019026 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:25.018964 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:25.019026 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:25.018989 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585dd8c8cc-f2d22: secret "image-registry-tls" not found Jul 14 18:22:25.019103 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:25.019062 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls podName:96f143d8-874e-4d12-94c5-51227b2d0740 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.019038059 +0000 UTC m=+41.076706086 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls") pod "image-registry-585dd8c8cc-f2d22" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740") : secret "image-registry-tls" not found Jul 14 18:22:25.119561 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:25.119524 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:25.119764 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:25.119696 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:25.119764 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:25.119761 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls podName:5a06092a-5be1-49bf-98d3-4f47ae464faf nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.119746799 +0000 UTC m=+41.177414830 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls") pod "dns-default-hnl7l" (UID: "5a06092a-5be1-49bf-98d3-4f47ae464faf") : secret "dns-default-metrics-tls" not found Jul 14 18:22:27.033573 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.033552 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:27.033855 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.033701 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:27.033855 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.033721 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585dd8c8cc-f2d22: secret "image-registry-tls" not found Jul 14 18:22:27.033855 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.033775 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls podName:96f143d8-874e-4d12-94c5-51227b2d0740 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:31.033757703 +0000 UTC m=+45.091425731 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls") pod "image-registry-585dd8c8cc-f2d22" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740") : secret "image-registry-tls" not found Jul 14 18:22:27.079356 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.079324 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-l75ch"] Jul 14 18:22:27.117219 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.117185 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l75ch"] Jul 14 18:22:27.117391 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.117335 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.119783 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.119762 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 14 18:22:27.119783 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.119774 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 14 18:22:27.120152 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.120125 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 14 18:22:27.120152 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.120126 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 14 18:22:27.120792 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.120775 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vtlhx\"" Jul 14 18:22:27.134510 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.134478 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:27.134612 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.134598 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:27.134659 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.134655 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls podName:5a06092a-5be1-49bf-98d3-4f47ae464faf nodeName:}" failed. No retries permitted until 2026-07-14 18:22:31.134642056 +0000 UTC m=+45.192310083 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls") pod "dns-default-hnl7l" (UID: "5a06092a-5be1-49bf-98d3-4f47ae464faf") : secret "dns-default-metrics-tls" not found Jul 14 18:22:27.235220 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.235181 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rt76\" (UniqueName: \"kubernetes.io/projected/975f0279-7e84-48f1-932c-739390863380-kube-api-access-4rt76\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.235220 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.235222 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/975f0279-7e84-48f1-932c-739390863380-data-volume\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.235448 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.235282 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/975f0279-7e84-48f1-932c-739390863380-crio-socket\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.235448 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.235350 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.235448 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.235401 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.335926 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.335834 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rt76\" (UniqueName: \"kubernetes.io/projected/975f0279-7e84-48f1-932c-739390863380-kube-api-access-4rt76\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.335926 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.335880 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/975f0279-7e84-48f1-932c-739390863380-data-volume\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.336133 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.336007 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/975f0279-7e84-48f1-932c-739390863380-crio-socket\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.336133 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.336123 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.336201 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.336164 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.336297 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.336238 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/975f0279-7e84-48f1-932c-739390863380-crio-socket\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.336297 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.336240 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:27.336440 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.336314 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.336440 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.336346 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.836331861 +0000 UTC m=+41.893999891 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : configmap "kube-rbac-proxy" not found Jul 14 18:22:27.336440 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.336242 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/975f0279-7e84-48f1-932c-739390863380-data-volume\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.336440 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.336400 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:27.836387319 +0000 UTC m=+41.894055354 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.348786 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.348759 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rt76\" (UniqueName: \"kubernetes.io/projected/975f0279-7e84-48f1-932c-739390863380-kube-api-access-4rt76\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.651477 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.651392 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-z5d6b" event={"ID":"0cf86795-02bf-4e52-a5ac-ae404d64f9bd","Type":"ContainerStarted","Data":"db97cf86cb3afe563f408977c1c5e92cfa97cea5310218334ddeccadec14c4da"} Jul 14 18:22:27.702919 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.702867 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-z5d6b" podStartSLOduration=1.5262171119999999 podStartE2EDuration="4.702848908s" totalImagesPullingTime="3.176631796s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:27.702198629 +0000 UTC m=+41.759866678" watchObservedRunningTime="2026-07-14 18:22:27.702848908 +0000 UTC m=+41.760516957" Jul 14 18:22:27.839892 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.839851 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.839892 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:27.839898 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:27.840116 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.839983 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:27.840116 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.840022 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:27.840116 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.840048 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:28.840031122 +0000 UTC m=+42.897699149 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : configmap "kube-rbac-proxy" not found Jul 14 18:22:27.840116 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:27.840082 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:28.84006846 +0000 UTC m=+42.897736490 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:28.846741 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:28.846703 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:28.847177 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:28.846749 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:28.847177 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:28.846842 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:28.847177 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:28.846896 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:28.847177 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:28.846913 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:30.846895144 +0000 UTC m=+44.904563186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : configmap "kube-rbac-proxy" not found Jul 14 18:22:28.847177 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:28.846948 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:30.846932997 +0000 UTC m=+44.904601023 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:29.052078 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.052045 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-swtrt_11223d59-522e-4311-ad5c-fcc84cf423e8/dns-node-resolver/0.log" Jul 14 18:22:29.123971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.123891 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-9t6rt"] Jul 14 18:22:29.125608 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.125593 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:29.128198 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.128171 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hbd9g\"" Jul 14 18:22:29.128438 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.128296 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 14 18:22:29.128438 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.128314 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 14 18:22:29.138911 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.138884 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-9t6rt"] Jul 14 18:22:29.250053 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.249992 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/343af867-9336-435a-8a69-2f3365f6798e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:29.250243 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.250200 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:29.350948 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.350904 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:29.351110 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.350960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/343af867-9336-435a-8a69-2f3365f6798e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:29.351110 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:29.351070 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:22:29.351172 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:29.351136 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert podName:343af867-9336-435a-8a69-2f3365f6798e nodeName:}" failed. No retries permitted until 2026-07-14 18:22:29.851119751 +0000 UTC m=+43.908787779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9t6rt" (UID: "343af867-9336-435a-8a69-2f3365f6798e") : secret "networking-console-plugin-cert" not found Jul 14 18:22:29.351891 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.351870 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/343af867-9336-435a-8a69-2f3365f6798e-nginx-conf\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:29.852356 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.852323 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hbw2t_0a9e50d0-df10-47d4-9ad8-717bc1af9150/node-ca/0.log" Jul 14 18:22:29.854663 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:29.854638 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:29.854765 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:29.854752 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:22:29.854821 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:29.854812 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert podName:343af867-9336-435a-8a69-2f3365f6798e nodeName:}" failed. No retries permitted until 2026-07-14 18:22:30.854798925 +0000 UTC m=+44.912466952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9t6rt" (UID: "343af867-9336-435a-8a69-2f3365f6798e") : secret "networking-console-plugin-cert" not found Jul 14 18:22:30.863519 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:30.863465 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:30.863519 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:30.863522 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:30.864043 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:30.863581 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:30.864043 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:30.863630 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:30.864043 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:30.863678 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:22:30.864043 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:30.863698 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:34.863677031 +0000 UTC m=+48.921345061 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:30.864043 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:30.863719 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert podName:343af867-9336-435a-8a69-2f3365f6798e nodeName:}" failed. No retries permitted until 2026-07-14 18:22:32.8637077 +0000 UTC m=+46.921375726 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9t6rt" (UID: "343af867-9336-435a-8a69-2f3365f6798e") : secret "networking-console-plugin-cert" not found Jul 14 18:22:30.864043 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:30.863744 2569 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 14 18:22:30.864043 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:30.863772 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:34.86376443 +0000 UTC m=+48.921432456 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : configmap "kube-rbac-proxy" not found Jul 14 18:22:31.052292 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:31.052223 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z5d6b_0cf86795-02bf-4e52-a5ac-ae404d64f9bd/serve-healthcheck-canary/0.log" Jul 14 18:22:31.065414 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:31.065365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:31.065541 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:31.065524 2569 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 14 18:22:31.065577 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:31.065544 2569 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-585dd8c8cc-f2d22: secret "image-registry-tls" not found Jul 14 18:22:31.065609 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:31.065600 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls podName:96f143d8-874e-4d12-94c5-51227b2d0740 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:39.065584683 +0000 UTC m=+53.123252709 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls") pod "image-registry-585dd8c8cc-f2d22" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740") : secret "image-registry-tls" not found Jul 14 18:22:31.166156 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:31.166043 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:31.166374 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:31.166221 2569 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 14 18:22:31.166374 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:31.166329 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls podName:5a06092a-5be1-49bf-98d3-4f47ae464faf nodeName:}" failed. No retries permitted until 2026-07-14 18:22:39.166306545 +0000 UTC m=+53.223974571 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls") pod "dns-default-hnl7l" (UID: "5a06092a-5be1-49bf-98d3-4f47ae464faf") : secret "dns-default-metrics-tls" not found Jul 14 18:22:32.879172 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:32.879119 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:32.879635 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:32.879282 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:22:32.879635 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:32.879341 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert podName:343af867-9336-435a-8a69-2f3365f6798e nodeName:}" failed. No retries permitted until 2026-07-14 18:22:36.879325455 +0000 UTC m=+50.936993481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9t6rt" (UID: "343af867-9336-435a-8a69-2f3365f6798e") : secret "networking-console-plugin-cert" not found Jul 14 18:22:34.895610 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:34.895571 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:34.896035 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:34.895683 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:34.896035 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:34.895742 2569 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 14 18:22:34.896035 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:34.895812 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls podName:975f0279-7e84-48f1-932c-739390863380 nodeName:}" failed. No retries permitted until 2026-07-14 18:22:42.895796484 +0000 UTC m=+56.953464512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls") pod "insights-runtime-extractor-l75ch" (UID: "975f0279-7e84-48f1-932c-739390863380") : secret "insights-runtime-extractor-tls" not found Jul 14 18:22:34.896149 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:34.896137 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/975f0279-7e84-48f1-932c-739390863380-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:36.913804 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:36.913748 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:36.914383 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:36.913953 2569 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 14 18:22:36.914383 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:22:36.914072 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert podName:343af867-9336-435a-8a69-2f3365f6798e nodeName:}" failed. No retries permitted until 2026-07-14 18:22:44.914022731 +0000 UTC m=+58.971690759 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-9t6rt" (UID: "343af867-9336-435a-8a69-2f3365f6798e") : secret "networking-console-plugin-cert" not found Jul 14 18:22:39.131800 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.131764 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:39.134839 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.134806 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"image-registry-585dd8c8cc-f2d22\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:39.226067 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.226028 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:39.232489 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.232460 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:39.234804 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.234769 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/5a06092a-5be1-49bf-98d3-4f47ae464faf-metrics-tls\") pod \"dns-default-hnl7l\" (UID: \"5a06092a-5be1-49bf-98d3-4f47ae464faf\") " pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:39.268722 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.268684 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:39.384655 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.384539 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-585dd8c8cc-f2d22"] Jul 14 18:22:39.387869 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:39.387845 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96f143d8_874e_4d12_94c5_51227b2d0740.slice/crio-0ac49d36c57e361a05ccee8417259a021b90e84e6be74cd3cf7a749b4d771b23 WatchSource:0}: Error finding container 0ac49d36c57e361a05ccee8417259a021b90e84e6be74cd3cf7a749b4d771b23: Status 404 returned error can't find the container with id 0ac49d36c57e361a05ccee8417259a021b90e84e6be74cd3cf7a749b4d771b23 Jul 14 18:22:39.424131 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.424103 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hnl7l"] Jul 14 18:22:39.431325 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:39.431293 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a06092a_5be1_49bf_98d3_4f47ae464faf.slice/crio-cb5abfde99f4ce6c71e405ac886b65a838ed2cd6c80c2e16778467708896e6d9 WatchSource:0}: Error finding container cb5abfde99f4ce6c71e405ac886b65a838ed2cd6c80c2e16778467708896e6d9: Status 404 returned error can't find the container with id cb5abfde99f4ce6c71e405ac886b65a838ed2cd6c80c2e16778467708896e6d9 Jul 14 18:22:39.683394 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.683295 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" event={"ID":"96f143d8-874e-4d12-94c5-51227b2d0740","Type":"ContainerStarted","Data":"fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd"} Jul 14 18:22:39.683394 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.683355 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" event={"ID":"96f143d8-874e-4d12-94c5-51227b2d0740","Type":"ContainerStarted","Data":"0ac49d36c57e361a05ccee8417259a021b90e84e6be74cd3cf7a749b4d771b23"} Jul 14 18:22:39.683394 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.683381 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:22:39.684438 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.684408 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hnl7l" event={"ID":"5a06092a-5be1-49bf-98d3-4f47ae464faf","Type":"ContainerStarted","Data":"cb5abfde99f4ce6c71e405ac886b65a838ed2cd6c80c2e16778467708896e6d9"} Jul 14 18:22:39.739499 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:39.739453 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" podStartSLOduration=37.739435869 podStartE2EDuration="37.739435869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:39.739173818 +0000 UTC m=+53.796841867" watchObservedRunningTime="2026-07-14 18:22:39.739435869 +0000 UTC m=+53.797103895" Jul 14 18:22:41.691321 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:41.691291 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hnl7l" event={"ID":"5a06092a-5be1-49bf-98d3-4f47ae464faf","Type":"ContainerStarted","Data":"89794719448f5f2e708e7557b57a1fd7e021dd072352a5566489bb7798a2f453"} Jul 14 18:22:42.695856 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:42.695822 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hnl7l" event={"ID":"5a06092a-5be1-49bf-98d3-4f47ae464faf","Type":"ContainerStarted","Data":"818854a6d910df9295862c018d91b64a4fc9c4c4c951117bb6e4d509442a21bc"} Jul 14 18:22:42.696355 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:42.695948 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:42.718455 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:42.718401 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hnl7l" podStartSLOduration=17.593758346 podStartE2EDuration="19.718382491s" totalImagesPullingTime="2.124624145s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:42.718051218 +0000 UTC m=+56.775719287" watchObservedRunningTime="2026-07-14 18:22:42.718382491 +0000 UTC m=+56.776050543" Jul 14 18:22:42.962828 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:42.962738 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:42.964943 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:42.964919 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/975f0279-7e84-48f1-932c-739390863380-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-l75ch\" (UID: \"975f0279-7e84-48f1-932c-739390863380\") " pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:43.026070 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:43.026036 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-l75ch" Jul 14 18:22:43.189126 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:43.189099 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-l75ch"] Jul 14 18:22:43.191645 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:43.191606 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod975f0279_7e84_48f1_932c_739390863380.slice/crio-d131f76d6e150f9ada3c05c4cc0c42b33d91a252c6969f869398cecb3e81a8a5 WatchSource:0}: Error finding container d131f76d6e150f9ada3c05c4cc0c42b33d91a252c6969f869398cecb3e81a8a5: Status 404 returned error can't find the container with id d131f76d6e150f9ada3c05c4cc0c42b33d91a252c6969f869398cecb3e81a8a5 Jul 14 18:22:43.701233 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:43.701181 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l75ch" event={"ID":"975f0279-7e84-48f1-932c-739390863380","Type":"ContainerStarted","Data":"d131f76d6e150f9ada3c05c4cc0c42b33d91a252c6969f869398cecb3e81a8a5"} Jul 14 18:22:44.979076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:44.979032 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:44.981656 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:44.981630 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/343af867-9336-435a-8a69-2f3365f6798e-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-9t6rt\" (UID: \"343af867-9336-435a-8a69-2f3365f6798e\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:45.035529 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:45.035494 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" Jul 14 18:22:47.401626 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:47.401594 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-9t6rt"] Jul 14 18:22:47.405536 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:47.405503 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod343af867_9336_435a_8a69_2f3365f6798e.slice/crio-38124a719518e7047904baaf68f36d3c1f59ed51b99a61792b17b39cc763ef31 WatchSource:0}: Error finding container 38124a719518e7047904baaf68f36d3c1f59ed51b99a61792b17b39cc763ef31: Status 404 returned error can't find the container with id 38124a719518e7047904baaf68f36d3c1f59ed51b99a61792b17b39cc763ef31 Jul 14 18:22:47.713946 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:47.713854 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l75ch" event={"ID":"975f0279-7e84-48f1-932c-739390863380","Type":"ContainerStarted","Data":"d388b56be7d2a54df545f6f396f03632c03e9c290cc2419736b7d6fef2f7fb57"} Jul 14 18:22:47.714917 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:47.714892 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" event={"ID":"343af867-9336-435a-8a69-2f3365f6798e","Type":"ContainerStarted","Data":"38124a719518e7047904baaf68f36d3c1f59ed51b99a61792b17b39cc763ef31"} Jul 14 18:22:48.720434 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:48.720324 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" event={"ID":"343af867-9336-435a-8a69-2f3365f6798e","Type":"ContainerStarted","Data":"451c4b0606f6bce391b0bc53d330a67692d5ca7b81f30557f717c20901482e1c"} Jul 14 18:22:48.722500 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:48.722474 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l75ch" event={"ID":"975f0279-7e84-48f1-932c-739390863380","Type":"ContainerStarted","Data":"299a3144a2baceae32fafefcad6d210ebe1521f4ad125123c7ce35902aff29f6"} Jul 14 18:22:49.634280 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:49.634237 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k8sz5" Jul 14 18:22:49.686522 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:49.686466 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-9t6rt" podStartSLOduration=19.71769228 podStartE2EDuration="20.686442894s" totalImagesPullingTime="968.750614ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:48.753982796 +0000 UTC m=+62.811650863" watchObservedRunningTime="2026-07-14 18:22:49.686442894 +0000 UTC m=+63.744110947" Jul 14 18:22:50.731646 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:50.731607 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-l75ch" event={"ID":"975f0279-7e84-48f1-932c-739390863380","Type":"ContainerStarted","Data":"a0053b7cbf3056f57a20fcec9456635b1e8e2371c723d9f605b5e41f0c897e4d"} Jul 14 18:22:50.757817 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:50.757771 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-l75ch" podStartSLOduration=16.560711194 podStartE2EDuration="23.757756582s" totalImagesPullingTime="7.197045388s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:22:27 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:50.757602481 +0000 UTC m=+64.815270530" watchObservedRunningTime="2026-07-14 18:22:50.757756582 +0000 UTC m=+64.815424631" Jul 14 18:22:52.138598 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.138556 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:52.138598 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.138598 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:52.141961 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.141941 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 14 18:22:52.142917 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.142903 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 14 18:22:52.151491 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.151467 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/70bf487c-b158-4aff-9aa0-bc83b031e407-original-pull-secret\") pod \"global-pull-secret-syncer-rx2bj\" (UID: \"70bf487c-b158-4aff-9aa0-bc83b031e407\") " pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:52.153737 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.153705 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 14 18:22:52.161981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.161954 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cgmd\" (UniqueName: \"kubernetes.io/projected/cb3370dd-1592-4848-b5da-9e27330cd28c-kube-api-access-6cgmd\") pod \"network-check-target-nlzhr\" (UID: \"cb3370dd-1592-4848-b5da-9e27330cd28c\") " pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:52.238946 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.238910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:52.242457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.242434 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 14 18:22:52.252076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.252044 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/50318283-8577-410a-af69-d1f7a903a1c3-metrics-certs\") pod \"network-metrics-daemon-6dr2s\" (UID: \"50318283-8577-410a-af69-d1f7a903a1c3\") " pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:52.288845 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.288815 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hzvpc\"" Jul 14 18:22:52.295720 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.295699 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dr2s" Jul 14 18:22:52.394716 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.394646 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lzfqw\"" Jul 14 18:22:52.399103 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.399078 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-rx2bj" Jul 14 18:22:52.402416 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.402390 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:52.426328 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.426294 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6dr2s"] Jul 14 18:22:52.428724 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:52.428680 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50318283_8577_410a_af69_d1f7a903a1c3.slice/crio-69876698a7eac93205551b9544723b10894a14934490d3ab572bec2f8c6e0c76 WatchSource:0}: Error finding container 69876698a7eac93205551b9544723b10894a14934490d3ab572bec2f8c6e0c76: Status 404 returned error can't find the container with id 69876698a7eac93205551b9544723b10894a14934490d3ab572bec2f8c6e0c76 Jul 14 18:22:52.542638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.542607 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-rx2bj"] Jul 14 18:22:52.545707 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:52.545671 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70bf487c_b158_4aff_9aa0_bc83b031e407.slice/crio-ed674bba2c6f730f64c5972e46a51a8bfebfe02a219cf89de0e6c0056b4b81d1 WatchSource:0}: Error finding container ed674bba2c6f730f64c5972e46a51a8bfebfe02a219cf89de0e6c0056b4b81d1: Status 404 returned error can't find the container with id ed674bba2c6f730f64c5972e46a51a8bfebfe02a219cf89de0e6c0056b4b81d1 Jul 14 18:22:52.565302 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.565278 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nlzhr"] Jul 14 18:22:52.567806 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:52.567784 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3370dd_1592_4848_b5da_9e27330cd28c.slice/crio-28a67afb6d6b6831f948c5eaa4c257b1f156c1d41cce79577a02c91029a520a8 WatchSource:0}: Error finding container 28a67afb6d6b6831f948c5eaa4c257b1f156c1d41cce79577a02c91029a520a8: Status 404 returned error can't find the container with id 28a67afb6d6b6831f948c5eaa4c257b1f156c1d41cce79577a02c91029a520a8 Jul 14 18:22:52.702970 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.702891 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hnl7l" Jul 14 18:22:52.740325 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.740289 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nlzhr" event={"ID":"cb3370dd-1592-4848-b5da-9e27330cd28c","Type":"ContainerStarted","Data":"28a67afb6d6b6831f948c5eaa4c257b1f156c1d41cce79577a02c91029a520a8"} Jul 14 18:22:52.741320 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.741295 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6dr2s" event={"ID":"50318283-8577-410a-af69-d1f7a903a1c3","Type":"ContainerStarted","Data":"69876698a7eac93205551b9544723b10894a14934490d3ab572bec2f8c6e0c76"} Jul 14 18:22:52.742246 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:52.742226 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rx2bj" event={"ID":"70bf487c-b158-4aff-9aa0-bc83b031e407","Type":"ContainerStarted","Data":"ed674bba2c6f730f64c5972e46a51a8bfebfe02a219cf89de0e6c0056b4b81d1"} Jul 14 18:22:54.347621 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.347561 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-585dd8c8cc-f2d22"] Jul 14 18:22:54.349047 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.348637 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm"] Jul 14 18:22:54.353008 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.352984 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" Jul 14 18:22:54.355150 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.355117 2569 patch_prober.go:28] interesting pod/image-registry-585dd8c8cc-f2d22 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 14 18:22:54.355302 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.355176 2569 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" podUID="96f143d8-874e-4d12-94c5-51227b2d0740" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 14 18:22:54.368689 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.368464 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 14 18:22:54.368689 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.368521 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w"] Jul 14 18:22:54.368895 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.368772 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-wp4ws\"" Jul 14 18:22:54.369514 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.369227 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 14 18:22:54.370124 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.369777 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 14 18:22:54.370124 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.369981 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 14 18:22:54.371833 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.371343 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.376651 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.376508 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 14 18:22:54.406972 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.406898 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w"] Jul 14 18:22:54.420534 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.420300 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm"] Jul 14 18:22:54.457022 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.456974 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ffecb070-166f-4df2-b7c7-992cede52b5e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78ffb7944b-nvwvm\" (UID: \"ffecb070-166f-4df2-b7c7-992cede52b5e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" Jul 14 18:22:54.457022 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.457024 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbhd8\" (UniqueName: \"kubernetes.io/projected/ffecb070-166f-4df2-b7c7-992cede52b5e-kube-api-access-sbhd8\") pod \"managed-serviceaccount-addon-agent-78ffb7944b-nvwvm\" (UID: \"ffecb070-166f-4df2-b7c7-992cede52b5e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" Jul 14 18:22:54.457387 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.457077 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.457387 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.457115 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68dnp\" (UniqueName: \"kubernetes.io/projected/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-kube-api-access-68dnp\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.457387 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.457155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-tmp\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.458174 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.458148 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-558f9b9449-ccgtp"] Jul 14 18:22:54.460312 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.460284 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.497660 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.497631 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558f9b9449-ccgtp"] Jul 14 18:22:54.558442 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558404 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ffecb070-166f-4df2-b7c7-992cede52b5e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78ffb7944b-nvwvm\" (UID: \"ffecb070-166f-4df2-b7c7-992cede52b5e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" Jul 14 18:22:54.558627 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558447 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.558627 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e14c8b14-5897-48fb-8226-1d6ae22c52a1-image-registry-private-configuration\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.558627 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558509 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-tmp\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.558627 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558527 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e14c8b14-5897-48fb-8226-1d6ae22c52a1-installation-pull-secrets\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.558627 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558545 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz5tw\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-kube-api-access-lz5tw\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.558627 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558577 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e14c8b14-5897-48fb-8226-1d6ae22c52a1-trusted-ca\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.558627 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558612 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbhd8\" (UniqueName: \"kubernetes.io/projected/ffecb070-166f-4df2-b7c7-992cede52b5e-kube-api-access-sbhd8\") pod \"managed-serviceaccount-addon-agent-78ffb7944b-nvwvm\" (UID: \"ffecb070-166f-4df2-b7c7-992cede52b5e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" Jul 14 18:22:54.558984 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558665 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-bound-sa-token\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.558984 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558697 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68dnp\" (UniqueName: \"kubernetes.io/projected/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-kube-api-access-68dnp\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.558984 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558757 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-registry-tls\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.558984 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558797 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e14c8b14-5897-48fb-8226-1d6ae22c52a1-registry-certificates\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.558984 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.558843 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e14c8b14-5897-48fb-8226-1d6ae22c52a1-ca-trust-extracted\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.559239 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.559204 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-tmp\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.561492 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.561416 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-klusterlet-config\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.561492 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.561421 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ffecb070-166f-4df2-b7c7-992cede52b5e-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-78ffb7944b-nvwvm\" (UID: \"ffecb070-166f-4df2-b7c7-992cede52b5e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" Jul 14 18:22:54.635632 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.635550 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbhd8\" (UniqueName: \"kubernetes.io/projected/ffecb070-166f-4df2-b7c7-992cede52b5e-kube-api-access-sbhd8\") pod \"managed-serviceaccount-addon-agent-78ffb7944b-nvwvm\" (UID: \"ffecb070-166f-4df2-b7c7-992cede52b5e\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" Jul 14 18:22:54.645325 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.645291 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68dnp\" (UniqueName: \"kubernetes.io/projected/436b96d6-e3f7-4ec1-b570-30bfab7b55cb-kube-api-access-68dnp\") pod \"klusterlet-addon-workmgr-578ff75b-mgz2w\" (UID: \"436b96d6-e3f7-4ec1-b570-30bfab7b55cb\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.660150 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660111 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e14c8b14-5897-48fb-8226-1d6ae22c52a1-image-registry-private-configuration\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.660340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660171 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e14c8b14-5897-48fb-8226-1d6ae22c52a1-installation-pull-secrets\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.660340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660204 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz5tw\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-kube-api-access-lz5tw\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.660340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660228 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e14c8b14-5897-48fb-8226-1d6ae22c52a1-trusted-ca\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.660340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660280 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-bound-sa-token\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.660340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660313 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-registry-tls\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.660340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660339 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e14c8b14-5897-48fb-8226-1d6ae22c52a1-registry-certificates\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.660649 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660369 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e14c8b14-5897-48fb-8226-1d6ae22c52a1-ca-trust-extracted\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.660842 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.660754 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/e14c8b14-5897-48fb-8226-1d6ae22c52a1-ca-trust-extracted\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.661649 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.661615 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/e14c8b14-5897-48fb-8226-1d6ae22c52a1-registry-certificates\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.662168 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.662141 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/e14c8b14-5897-48fb-8226-1d6ae22c52a1-trusted-ca\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.663335 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.663302 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/e14c8b14-5897-48fb-8226-1d6ae22c52a1-image-registry-private-configuration\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.663422 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.663296 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/e14c8b14-5897-48fb-8226-1d6ae22c52a1-installation-pull-secrets\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.663772 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.663751 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-registry-tls\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.673473 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.673449 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" Jul 14 18:22:54.685810 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.685711 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:22:54.706825 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.706782 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz5tw\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-kube-api-access-lz5tw\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.711177 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.711136 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e14c8b14-5897-48fb-8226-1d6ae22c52a1-bound-sa-token\") pod \"image-registry-558f9b9449-ccgtp\" (UID: \"e14c8b14-5897-48fb-8226-1d6ae22c52a1\") " pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.752962 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.752926 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6dr2s" event={"ID":"50318283-8577-410a-af69-d1f7a903a1c3","Type":"ContainerStarted","Data":"6c0b45e81b02a4996877530afb543a582eff0644c1dc4ef85090198772272be5"} Jul 14 18:22:54.753144 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.753019 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6dr2s" event={"ID":"50318283-8577-410a-af69-d1f7a903a1c3","Type":"ContainerStarted","Data":"94caced385d91b0eff700a23c1c74a27550fa69627b01b42b9a1090ffaede60a"} Jul 14 18:22:54.772405 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.772370 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:22:54.831875 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:54.831822 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6dr2s" podStartSLOduration=66.408736312 podStartE2EDuration="1m7.831801431s" totalImagesPullingTime="1.423065119s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:54.830958903 +0000 UTC m=+68.888626953" watchObservedRunningTime="2026-07-14 18:22:54.831801431 +0000 UTC m=+68.889469482" Jul 14 18:22:55.841732 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:55.841701 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm"] Jul 14 18:22:55.862008 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:55.861971 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-558f9b9449-ccgtp"] Jul 14 18:22:56.062483 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:56.062449 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w"] Jul 14 18:22:56.764700 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:56.764665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nlzhr" event={"ID":"cb3370dd-1592-4848-b5da-9e27330cd28c","Type":"ContainerStarted","Data":"17b45829513860361f8ae0aa47cc105be8b2edab4b488523a2b7e78bda078800"} Jul 14 18:22:56.764894 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:56.764787 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:22:56.787420 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:56.787366 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nlzhr" podStartSLOduration=66.664359766 podStartE2EDuration="1m9.787347468s" totalImagesPullingTime="3.122987702s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:56.786646503 +0000 UTC m=+70.844314554" watchObservedRunningTime="2026-07-14 18:22:56.787347468 +0000 UTC m=+70.845015517" Jul 14 18:22:57.213852 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:57.213760 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podffecb070_166f_4df2_b7c7_992cede52b5e.slice/crio-54719974b3457fa3610dcf1158c6df06fd6ff5dc6427c9b7dffac1d111f6e9a4 WatchSource:0}: Error finding container 54719974b3457fa3610dcf1158c6df06fd6ff5dc6427c9b7dffac1d111f6e9a4: Status 404 returned error can't find the container with id 54719974b3457fa3610dcf1158c6df06fd6ff5dc6427c9b7dffac1d111f6e9a4 Jul 14 18:22:57.214420 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:57.214310 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod436b96d6_e3f7_4ec1_b570_30bfab7b55cb.slice/crio-45382614cb6b2d2378af5ec9c2394813296462c2f8238d27baa5fe3b70a9438a WatchSource:0}: Error finding container 45382614cb6b2d2378af5ec9c2394813296462c2f8238d27baa5fe3b70a9438a: Status 404 returned error can't find the container with id 45382614cb6b2d2378af5ec9c2394813296462c2f8238d27baa5fe3b70a9438a Jul 14 18:22:57.215267 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:22:57.215226 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode14c8b14_5897_48fb_8226_1d6ae22c52a1.slice/crio-e4499b60a70230546de324f5e556132618c54e97911d4ef72b26bcf0c8ed9647 WatchSource:0}: Error finding container e4499b60a70230546de324f5e556132618c54e97911d4ef72b26bcf0c8ed9647: Status 404 returned error can't find the container with id e4499b60a70230546de324f5e556132618c54e97911d4ef72b26bcf0c8ed9647 Jul 14 18:22:57.772327 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:57.772282 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" event={"ID":"436b96d6-e3f7-4ec1-b570-30bfab7b55cb","Type":"ContainerStarted","Data":"45382614cb6b2d2378af5ec9c2394813296462c2f8238d27baa5fe3b70a9438a"} Jul 14 18:22:57.774233 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:57.774180 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-rx2bj" event={"ID":"70bf487c-b158-4aff-9aa0-bc83b031e407","Type":"ContainerStarted","Data":"ecf5f252ae60151fe6ac80bad193770f1dfbf8949ee00ad1f428862ed3cafde5"} Jul 14 18:22:57.775708 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:57.775679 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" event={"ID":"ffecb070-166f-4df2-b7c7-992cede52b5e","Type":"ContainerStarted","Data":"54719974b3457fa3610dcf1158c6df06fd6ff5dc6427c9b7dffac1d111f6e9a4"} Jul 14 18:22:57.777268 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:57.777221 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" event={"ID":"e14c8b14-5897-48fb-8226-1d6ae22c52a1","Type":"ContainerStarted","Data":"4c7ea4e0f2ccac3fc7582fd3db81a95fab22b76d7d275017a462312d94245a36"} Jul 14 18:22:57.777385 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:57.777286 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" event={"ID":"e14c8b14-5897-48fb-8226-1d6ae22c52a1","Type":"ContainerStarted","Data":"e4499b60a70230546de324f5e556132618c54e97911d4ef72b26bcf0c8ed9647"} Jul 14 18:22:57.822425 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:57.822367 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" podStartSLOduration=3.822345833 podStartE2EDuration="3.822345833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:57.821379875 +0000 UTC m=+71.879047942" watchObservedRunningTime="2026-07-14 18:22:57.822345833 +0000 UTC m=+71.880013881" Jul 14 18:22:57.822655 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:57.822624 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-rx2bj" podStartSLOduration=66.119433739 podStartE2EDuration="1m10.82261118s" totalImagesPullingTime="4.703177441s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 18:21:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:22:57.79394786 +0000 UTC m=+71.851615925" watchObservedRunningTime="2026-07-14 18:22:57.82261118 +0000 UTC m=+71.880279226" Jul 14 18:22:58.781539 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:22:58.781497 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:23:00.532002 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.531964 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x9qqw"] Jul 14 18:23:00.534650 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.534378 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.539593 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.539372 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\"" Jul 14 18:23:00.539593 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.539386 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\"" Jul 14 18:23:00.540978 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.540955 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-vq9nz\"" Jul 14 18:23:00.541079 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.540987 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 14 18:23:00.541234 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.541218 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 14 18:23:00.541335 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.541283 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 14 18:23:00.565747 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.565718 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x9qqw"] Jul 14 18:23:00.609395 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.609355 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7285a78-51a1-4e85-bffe-9544e843dafb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.609588 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.609423 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.609588 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.609485 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vkzw\" (UniqueName: \"kubernetes.io/projected/a7285a78-51a1-4e85-bffe-9544e843dafb-kube-api-access-6vkzw\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.609588 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.609524 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.710170 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.710131 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vkzw\" (UniqueName: \"kubernetes.io/projected/a7285a78-51a1-4e85-bffe-9544e843dafb-kube-api-access-6vkzw\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.710389 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.710190 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.710389 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.710226 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7285a78-51a1-4e85-bffe-9544e843dafb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.710389 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:23:00.710326 2569 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found Jul 14 18:23:00.710580 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.710301 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.711461 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:23:00.711438 2569 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-tls podName:a7285a78-51a1-4e85-bffe-9544e843dafb nodeName:}" failed. No retries permitted until 2026-07-14 18:23:01.211407233 +0000 UTC m=+75.269075267 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-tls") pod "prometheus-operator-64785b6675-x9qqw" (UID: "a7285a78-51a1-4e85-bffe-9544e843dafb") : secret "prometheus-operator-tls" not found Jul 14 18:23:00.712054 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.712020 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a7285a78-51a1-4e85-bffe-9544e843dafb-metrics-client-ca\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.715582 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.715118 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:00.724155 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:00.724001 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vkzw\" (UniqueName: \"kubernetes.io/projected/a7285a78-51a1-4e85-bffe-9544e843dafb-kube-api-access-6vkzw\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:01.216818 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:01.216788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:01.219530 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:01.219505 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/a7285a78-51a1-4e85-bffe-9544e843dafb-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-x9qqw\" (UID: \"a7285a78-51a1-4e85-bffe-9544e843dafb\") " pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:01.447738 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:01.447655 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" Jul 14 18:23:01.604666 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:01.604642 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-x9qqw"] Jul 14 18:23:01.607497 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:23:01.607461 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7285a78_51a1_4e85_bffe_9544e843dafb.slice/crio-88e4ab55ab638f0c24e80dcae546c2bebaec7193432b5ea18fd2216014a0a3bd WatchSource:0}: Error finding container 88e4ab55ab638f0c24e80dcae546c2bebaec7193432b5ea18fd2216014a0a3bd: Status 404 returned error can't find the container with id 88e4ab55ab638f0c24e80dcae546c2bebaec7193432b5ea18fd2216014a0a3bd Jul 14 18:23:01.794194 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:01.794140 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" event={"ID":"ffecb070-166f-4df2-b7c7-992cede52b5e","Type":"ContainerStarted","Data":"7be9059ffb48fa25529a25652b8d40a422bd2cf7f8761ace05cf3f9abfb7272d"} Jul 14 18:23:01.795553 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:01.795525 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" event={"ID":"a7285a78-51a1-4e85-bffe-9544e843dafb","Type":"ContainerStarted","Data":"88e4ab55ab638f0c24e80dcae546c2bebaec7193432b5ea18fd2216014a0a3bd"} Jul 14 18:23:01.815984 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:01.815916 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-78ffb7944b-nvwvm" podStartSLOduration=3.8364909430000003 podStartE2EDuration="7.815896111s" totalImagesPullingTime="3.979405168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:01.814164858 +0000 UTC m=+75.871832926" watchObservedRunningTime="2026-07-14 18:23:01.815896111 +0000 UTC m=+75.873564161" Jul 14 18:23:03.804329 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:03.804285 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" event={"ID":"436b96d6-e3f7-4ec1-b570-30bfab7b55cb","Type":"ContainerStarted","Data":"c454a03e2d527e86c2c7c46552b00d5d1f917a5f923ab1a856ee189a69f4755a"} Jul 14 18:23:03.804791 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:03.804359 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:23:03.806283 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:03.806243 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" Jul 14 18:23:03.806391 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:03.806304 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" event={"ID":"a7285a78-51a1-4e85-bffe-9544e843dafb","Type":"ContainerStarted","Data":"c4c8d56c0a600cc9fc6a2f4ea497a409c87dc42e6f0762da71a207f25d07b980"} Jul 14 18:23:03.806391 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:03.806328 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" event={"ID":"a7285a78-51a1-4e85-bffe-9544e843dafb","Type":"ContainerStarted","Data":"40f007d244a4685ace6380ea3acb9402e69635aa151f497d87ca97d9d57e2691"} Jul 14 18:23:03.839081 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:03.839015 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-578ff75b-mgz2w" podStartSLOduration=3.449985003 podStartE2EDuration="9.838993713s" totalImagesPullingTime="6.38900871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:22:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:03.83858175 +0000 UTC m=+77.896249800" watchObservedRunningTime="2026-07-14 18:23:03.838993713 +0000 UTC m=+77.896661767" Jul 14 18:23:03.888714 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:03.888616 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-x9qqw" podStartSLOduration=1.8991679430000001 podStartE2EDuration="3.888598024s" totalImagesPullingTime="1.989430081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:03.887697818 +0000 UTC m=+77.945365879" watchObservedRunningTime="2026-07-14 18:23:03.888598024 +0000 UTC m=+77.946266072" Jul 14 18:23:04.352067 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:04.352036 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:23:05.703168 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.703137 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b755759b9-5hrkk"] Jul 14 18:23:05.705300 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.705281 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.709246 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.709221 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 14 18:23:05.709349 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.709231 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 14 18:23:05.710236 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.710221 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-nsnnm\"" Jul 14 18:23:05.710346 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.710238 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 14 18:23:05.710670 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.710650 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 14 18:23:05.710763 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.710650 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 14 18:23:05.710823 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.710797 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 14 18:23:05.710909 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.710895 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 14 18:23:05.721795 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.721774 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b755759b9-5hrkk"] Jul 14 18:23:05.855344 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.855308 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-oauth-serving-cert\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.855344 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.855343 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hglz9\" (UniqueName: \"kubernetes.io/projected/540d9122-3c8e-4099-8cf8-713d9b0833f7-kube-api-access-hglz9\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.855516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.855366 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-service-ca\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.855516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.855453 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-config\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.855516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.855501 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-oauth-config\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.855630 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.855525 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-serving-cert\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.956223 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.956128 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-oauth-serving-cert\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.956223 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.956168 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hglz9\" (UniqueName: \"kubernetes.io/projected/540d9122-3c8e-4099-8cf8-713d9b0833f7-kube-api-access-hglz9\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.956223 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.956210 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-service-ca\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.956521 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.956282 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-config\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.956521 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.956331 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-oauth-config\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.956521 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.956365 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-serving-cert\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.956972 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.956942 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-oauth-serving-cert\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.957095 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.957014 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-config\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.957095 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.957015 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-service-ca\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.958791 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.958769 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-oauth-config\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.958902 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.958855 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-serving-cert\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.978207 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.978181 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hglz9\" (UniqueName: \"kubernetes.io/projected/540d9122-3c8e-4099-8cf8-713d9b0833f7-kube-api-access-hglz9\") pod \"console-5b755759b9-5hrkk\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:05.991717 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.991693 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-swldn"] Jul 14 18:23:05.994101 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.994083 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:05.999536 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.999513 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 14 18:23:05.999648 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.999515 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-h2g88\"" Jul 14 18:23:05.999648 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:05.999573 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 14 18:23:06.000241 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.000219 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 14 18:23:06.014594 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.014565 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:06.057394 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057364 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-textfile\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.057563 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057410 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-root\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.057563 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057430 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-wtmp\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.057563 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057478 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-sys\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.057563 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057515 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-tls\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.057693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057563 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.057693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057589 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0edeb880-60bd-454a-ad60-4debf10076f1-metrics-client-ca\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.057693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057610 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.057693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.057627 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz7pw\" (UniqueName: \"kubernetes.io/projected/0edeb880-60bd-454a-ad60-4debf10076f1-kube-api-access-sz7pw\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.147274 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.147232 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b755759b9-5hrkk"] Jul 14 18:23:06.149028 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:23:06.149002 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod540d9122_3c8e_4099_8cf8_713d9b0833f7.slice/crio-90ddda2b9238d20f6285ea383f9fbf07432dd1686994f9292a6f8002bd9d0f0c WatchSource:0}: Error finding container 90ddda2b9238d20f6285ea383f9fbf07432dd1686994f9292a6f8002bd9d0f0c: Status 404 returned error can't find the container with id 90ddda2b9238d20f6285ea383f9fbf07432dd1686994f9292a6f8002bd9d0f0c Jul 14 18:23:06.158819 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.158794 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.158928 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.158830 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz7pw\" (UniqueName: \"kubernetes.io/projected/0edeb880-60bd-454a-ad60-4debf10076f1-kube-api-access-sz7pw\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.158928 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.158909 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-textfile\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159030 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.158968 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-root\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159030 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.158989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-wtmp\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159030 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159005 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-sys\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159030 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159021 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-tls\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159194 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159062 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-root\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159194 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159070 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159194 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159079 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-sys\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159194 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159093 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0edeb880-60bd-454a-ad60-4debf10076f1-metrics-client-ca\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159194 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159142 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-wtmp\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159397 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159338 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-textfile\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159561 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159542 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0edeb880-60bd-454a-ad60-4debf10076f1-metrics-client-ca\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.159624 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.159612 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-accelerators-collector-config\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.161292 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.161271 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-tls\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.161366 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.161313 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0edeb880-60bd-454a-ad60-4debf10076f1-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.168647 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.168625 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz7pw\" (UniqueName: \"kubernetes.io/projected/0edeb880-60bd-454a-ad60-4debf10076f1-kube-api-access-sz7pw\") pod \"node-exporter-swldn\" (UID: \"0edeb880-60bd-454a-ad60-4debf10076f1\") " pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.303649 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.303617 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-swldn" Jul 14 18:23:06.313590 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:23:06.313562 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0edeb880_60bd_454a_ad60_4debf10076f1.slice/crio-11c01af3c28e3f54c55c725b4295d99d8aca7731f8b274292f8c68d540ac91ef WatchSource:0}: Error finding container 11c01af3c28e3f54c55c725b4295d99d8aca7731f8b274292f8c68d540ac91ef: Status 404 returned error can't find the container with id 11c01af3c28e3f54c55c725b4295d99d8aca7731f8b274292f8c68d540ac91ef Jul 14 18:23:06.817676 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.817634 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b755759b9-5hrkk" event={"ID":"540d9122-3c8e-4099-8cf8-713d9b0833f7","Type":"ContainerStarted","Data":"90ddda2b9238d20f6285ea383f9fbf07432dd1686994f9292a6f8002bd9d0f0c"} Jul 14 18:23:06.819075 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:06.819046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swldn" event={"ID":"0edeb880-60bd-454a-ad60-4debf10076f1","Type":"ContainerStarted","Data":"11c01af3c28e3f54c55c725b4295d99d8aca7731f8b274292f8c68d540ac91ef"} Jul 14 18:23:07.827325 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:07.827284 2569 generic.go:356] "Generic (PLEG): container finished" podID="0edeb880-60bd-454a-ad60-4debf10076f1" containerID="0678c355bc039dc4e1c871aa8566a535d9373104e24b117491a797dc85c7cce6" exitCode=0 Jul 14 18:23:07.827780 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:07.827387 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swldn" event={"ID":"0edeb880-60bd-454a-ad60-4debf10076f1","Type":"ContainerDied","Data":"0678c355bc039dc4e1c871aa8566a535d9373104e24b117491a797dc85c7cce6"} Jul 14 18:23:08.833848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:08.833811 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swldn" event={"ID":"0edeb880-60bd-454a-ad60-4debf10076f1","Type":"ContainerStarted","Data":"9391955fe71d071f9dc6cf359e3b2507705adc3160d77bc88f61ed428e5a0d22"} Jul 14 18:23:08.834339 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:08.833875 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-swldn" event={"ID":"0edeb880-60bd-454a-ad60-4debf10076f1","Type":"ContainerStarted","Data":"c479741168e09c41f216af5fcaca3b3e12beeb8418f401e425a357b94aed35a6"} Jul 14 18:23:08.869722 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:08.869663 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-swldn" podStartSLOduration=2.919778872 podStartE2EDuration="3.869640248s" totalImagesPullingTime="876.256798ms" totalInitContainerRuntime="73.604578ms" isStatefulPod=true podCreationTimestamp="2026-07-14 18:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:08.867333336 +0000 UTC m=+82.925001392" watchObservedRunningTime="2026-07-14 18:23:08.869640248 +0000 UTC m=+82.927308298" Jul 14 18:23:09.838962 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:09.838922 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b755759b9-5hrkk" event={"ID":"540d9122-3c8e-4099-8cf8-713d9b0833f7","Type":"ContainerStarted","Data":"d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90"} Jul 14 18:23:09.866769 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:09.866717 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b755759b9-5hrkk" podStartSLOduration=1.655501744 podStartE2EDuration="4.866699141s" totalImagesPullingTime="3.211197397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:09.865121041 +0000 UTC m=+83.922789546" watchObservedRunningTime="2026-07-14 18:23:09.866699141 +0000 UTC m=+83.924367190" Jul 14 18:23:10.457693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.457654 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-68ffd96d67-dmvng"] Jul 14 18:23:10.460046 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.460007 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.462919 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.462899 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Jul 14 18:23:10.463094 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.463078 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-vc6mw\"" Jul 14 18:23:10.463238 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.463218 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Jul 14 18:23:10.464285 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.464269 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 14 18:23:10.464565 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.464551 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-1bvn888lpedlo\"" Jul 14 18:23:10.464624 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.464585 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Jul 14 18:23:10.481099 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.481075 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68ffd96d67-dmvng"] Jul 14 18:23:10.602889 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.602846 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-client-ca-bundle\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.603068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.602898 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23c985fb-f138-4dda-a2e2-1f2b70bc8889-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.603068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.602935 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/23c985fb-f138-4dda-a2e2-1f2b70bc8889-audit-log\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.603068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.602965 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/23c985fb-f138-4dda-a2e2-1f2b70bc8889-metrics-server-audit-profiles\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.603068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.602999 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-secret-metrics-server-tls\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.603068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.603041 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngsr5\" (UniqueName: \"kubernetes.io/projected/23c985fb-f138-4dda-a2e2-1f2b70bc8889-kube-api-access-ngsr5\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.603230 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.603070 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-secret-metrics-server-client-certs\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.703789 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.703751 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/23c985fb-f138-4dda-a2e2-1f2b70bc8889-metrics-server-audit-profiles\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.703981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.703818 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-secret-metrics-server-tls\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.703981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.703848 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngsr5\" (UniqueName: \"kubernetes.io/projected/23c985fb-f138-4dda-a2e2-1f2b70bc8889-kube-api-access-ngsr5\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.703981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.703878 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-secret-metrics-server-client-certs\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.703981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.703934 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-client-ca-bundle\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.703981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.703967 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23c985fb-f138-4dda-a2e2-1f2b70bc8889-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.704232 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.704013 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/23c985fb-f138-4dda-a2e2-1f2b70bc8889-audit-log\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.704857 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.704828 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/23c985fb-f138-4dda-a2e2-1f2b70bc8889-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.704987 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.704900 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/23c985fb-f138-4dda-a2e2-1f2b70bc8889-metrics-server-audit-profiles\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.704987 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.704943 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/23c985fb-f138-4dda-a2e2-1f2b70bc8889-audit-log\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.706495 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.706472 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-secret-metrics-server-tls\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.706608 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.706591 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-secret-metrics-server-client-certs\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.706608 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.706600 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/23c985fb-f138-4dda-a2e2-1f2b70bc8889-client-ca-bundle\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.724757 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.724696 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngsr5\" (UniqueName: \"kubernetes.io/projected/23c985fb-f138-4dda-a2e2-1f2b70bc8889-kube-api-access-ngsr5\") pod \"metrics-server-68ffd96d67-dmvng\" (UID: \"23c985fb-f138-4dda-a2e2-1f2b70bc8889\") " pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.769850 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.769815 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:10.946021 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:10.945837 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-68ffd96d67-dmvng"] Jul 14 18:23:10.948616 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:23:10.948592 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c985fb_f138_4dda_a2e2_1f2b70bc8889.slice/crio-eb6ae8733b019187b8a38a01247a794d403ea62fca6a5cee9709a4259bd05cb7 WatchSource:0}: Error finding container eb6ae8733b019187b8a38a01247a794d403ea62fca6a5cee9709a4259bd05cb7: Status 404 returned error can't find the container with id eb6ae8733b019187b8a38a01247a794d403ea62fca6a5cee9709a4259bd05cb7 Jul 14 18:23:11.847615 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:11.847575 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" event={"ID":"23c985fb-f138-4dda-a2e2-1f2b70bc8889","Type":"ContainerStarted","Data":"eb6ae8733b019187b8a38a01247a794d403ea62fca6a5cee9709a4259bd05cb7"} Jul 14 18:23:12.853048 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:12.853011 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" event={"ID":"23c985fb-f138-4dda-a2e2-1f2b70bc8889","Type":"ContainerStarted","Data":"bea2add8241e3267f6a10df26bccee3a981079eaf5312632c5c36f2eb26a1bae"} Jul 14 18:23:12.874419 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:12.874358 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" podStartSLOduration=1.388075049 podStartE2EDuration="2.874334572s" totalImagesPullingTime="1.486259523s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:12.873579519 +0000 UTC m=+86.931247569" watchObservedRunningTime="2026-07-14 18:23:12.874334572 +0000 UTC m=+86.932002622" Jul 14 18:23:16.015422 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.015381 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:16.015839 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.015492 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:16.020391 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.020363 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:16.252419 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.252388 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-76ccf85775-dxvbp"] Jul 14 18:23:16.369646 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.369514 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76ccf85775-dxvbp"] Jul 14 18:23:16.369646 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.369616 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.377958 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.377926 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 14 18:23:16.457503 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.457468 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-service-ca\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.457503 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.457515 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-oauth-config\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.457730 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.457579 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-oauth-serving-cert\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.457730 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.457626 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-trusted-ca-bundle\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.457730 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.457676 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnhmr\" (UniqueName: \"kubernetes.io/projected/052da12e-bea0-4906-aae4-fcc958cae59d-kube-api-access-nnhmr\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.457730 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.457707 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-serving-cert\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.457871 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.457777 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-console-config\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.558660 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.558624 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnhmr\" (UniqueName: \"kubernetes.io/projected/052da12e-bea0-4906-aae4-fcc958cae59d-kube-api-access-nnhmr\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.558660 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.558663 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-serving-cert\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.558918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.558690 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-console-config\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.558918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.558713 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-service-ca\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.558918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.558730 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-oauth-config\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.558918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.558754 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-oauth-serving-cert\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.558918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.558775 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-trusted-ca-bundle\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.559613 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.559588 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-oauth-serving-cert\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.559739 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.559589 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-console-config\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.559739 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.559589 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-service-ca\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.559739 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.559687 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-trusted-ca-bundle\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.561600 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.561573 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-oauth-config\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.561703 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.561661 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-serving-cert\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.585844 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.585817 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnhmr\" (UniqueName: \"kubernetes.io/projected/052da12e-bea0-4906-aae4-fcc958cae59d-kube-api-access-nnhmr\") pod \"console-76ccf85775-dxvbp\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.679517 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.679427 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:16.858844 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.858794 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-76ccf85775-dxvbp"] Jul 14 18:23:16.861789 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:23:16.861762 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod052da12e_bea0_4906_aae4_fcc958cae59d.slice/crio-58fff18b1a5aebb0f3aac979cc6d1e47ca1604a50f1d696f61c9b4fcd9b31d38 WatchSource:0}: Error finding container 58fff18b1a5aebb0f3aac979cc6d1e47ca1604a50f1d696f61c9b4fcd9b31d38: Status 404 returned error can't find the container with id 58fff18b1a5aebb0f3aac979cc6d1e47ca1604a50f1d696f61c9b4fcd9b31d38 Jul 14 18:23:16.869443 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.869415 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76ccf85775-dxvbp" event={"ID":"052da12e-bea0-4906-aae4-fcc958cae59d","Type":"ContainerStarted","Data":"58fff18b1a5aebb0f3aac979cc6d1e47ca1604a50f1d696f61c9b4fcd9b31d38"} Jul 14 18:23:16.898695 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:16.898673 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:17.874897 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:17.874855 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76ccf85775-dxvbp" event={"ID":"052da12e-bea0-4906-aae4-fcc958cae59d","Type":"ContainerStarted","Data":"e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407"} Jul 14 18:23:17.919899 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:17.919851 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-76ccf85775-dxvbp" podStartSLOduration=1.9198347550000001 podStartE2EDuration="1.919834755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:23:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:23:17.919747075 +0000 UTC m=+91.977415125" watchObservedRunningTime="2026-07-14 18:23:17.919834755 +0000 UTC m=+91.977502803" Jul 14 18:23:19.378631 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.378583 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" podUID="96f143d8-874e-4d12-94c5-51227b2d0740" containerName="registry" containerID="cri-o://fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd" gracePeriod=30 Jul 14 18:23:19.623712 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.623686 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:23:19.784918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.784879 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") pod \"96f143d8-874e-4d12-94c5-51227b2d0740\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " Jul 14 18:23:19.785107 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.784935 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96f143d8-874e-4d12-94c5-51227b2d0740-ca-trust-extracted\") pod \"96f143d8-874e-4d12-94c5-51227b2d0740\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " Jul 14 18:23:19.785107 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.784970 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-bound-sa-token\") pod \"96f143d8-874e-4d12-94c5-51227b2d0740\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " Jul 14 18:23:19.785107 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.785000 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-registry-certificates\") pod \"96f143d8-874e-4d12-94c5-51227b2d0740\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " Jul 14 18:23:19.785107 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.785047 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-image-registry-private-configuration\") pod \"96f143d8-874e-4d12-94c5-51227b2d0740\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " Jul 14 18:23:19.785107 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.785071 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-installation-pull-secrets\") pod \"96f143d8-874e-4d12-94c5-51227b2d0740\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " Jul 14 18:23:19.785550 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.785115 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-kube-api-access-bkd4r\") pod \"96f143d8-874e-4d12-94c5-51227b2d0740\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " Jul 14 18:23:19.785550 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.785184 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-trusted-ca\") pod \"96f143d8-874e-4d12-94c5-51227b2d0740\" (UID: \"96f143d8-874e-4d12-94c5-51227b2d0740\") " Jul 14 18:23:19.785550 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.785449 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "96f143d8-874e-4d12-94c5-51227b2d0740" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:23:19.785953 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.785812 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "96f143d8-874e-4d12-94c5-51227b2d0740" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:23:19.788125 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.787996 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "96f143d8-874e-4d12-94c5-51227b2d0740" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:23:19.788125 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.788022 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "96f143d8-874e-4d12-94c5-51227b2d0740" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:23:19.788125 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.788101 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "96f143d8-874e-4d12-94c5-51227b2d0740" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:23:19.788373 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.788133 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "96f143d8-874e-4d12-94c5-51227b2d0740" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:23:19.788373 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.788191 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-kube-api-access-bkd4r" (OuterVolumeSpecName: "kube-api-access-bkd4r") pod "96f143d8-874e-4d12-94c5-51227b2d0740" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740"). InnerVolumeSpecName "kube-api-access-bkd4r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:23:19.790323 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.790304 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-558f9b9449-ccgtp" Jul 14 18:23:19.794139 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.794116 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/96f143d8-874e-4d12-94c5-51227b2d0740-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "96f143d8-874e-4d12-94c5-51227b2d0740" (UID: "96f143d8-874e-4d12-94c5-51227b2d0740"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:23:19.883215 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.883183 2569 generic.go:356] "Generic (PLEG): container finished" podID="96f143d8-874e-4d12-94c5-51227b2d0740" containerID="fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd" exitCode=0 Jul 14 18:23:19.883398 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.883232 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" Jul 14 18:23:19.883398 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.883270 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" event={"ID":"96f143d8-874e-4d12-94c5-51227b2d0740","Type":"ContainerDied","Data":"fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd"} Jul 14 18:23:19.883398 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.883323 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-585dd8c8cc-f2d22" event={"ID":"96f143d8-874e-4d12-94c5-51227b2d0740","Type":"ContainerDied","Data":"0ac49d36c57e361a05ccee8417259a021b90e84e6be74cd3cf7a749b4d771b23"} Jul 14 18:23:19.883398 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.883344 2569 scope.go:117] "RemoveContainer" containerID="fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd" Jul 14 18:23:19.886525 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.886500 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-trusted-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:19.886678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.886529 2569 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-registry-tls\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:19.886678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.886546 2569 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/96f143d8-874e-4d12-94c5-51227b2d0740-ca-trust-extracted\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:19.886678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.886563 2569 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-bound-sa-token\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:19.886678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.886578 2569 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/96f143d8-874e-4d12-94c5-51227b2d0740-registry-certificates\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:19.886678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.886594 2569 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-image-registry-private-configuration\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:19.886678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.886610 2569 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/96f143d8-874e-4d12-94c5-51227b2d0740-installation-pull-secrets\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:19.886678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.886630 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkd4r\" (UniqueName: \"kubernetes.io/projected/96f143d8-874e-4d12-94c5-51227b2d0740-kube-api-access-bkd4r\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:19.893084 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.893068 2569 scope.go:117] "RemoveContainer" containerID="fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd" Jul 14 18:23:19.893383 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:23:19.893362 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd\": container with ID starting with fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd not found: ID does not exist" containerID="fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd" Jul 14 18:23:19.893439 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.893391 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd"} err="failed to get container status \"fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd\": rpc error: code = NotFound desc = could not find container \"fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd\": container with ID starting with fc8ae553a23a27fb94ee501a10aa4af1482791086a4374ef7daa47b91510d9fd not found: ID does not exist" Jul 14 18:23:19.926431 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.926400 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-585dd8c8cc-f2d22"] Jul 14 18:23:19.940602 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:19.940570 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-585dd8c8cc-f2d22"] Jul 14 18:23:20.481465 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:20.481430 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96f143d8-874e-4d12-94c5-51227b2d0740" path="/var/lib/kubelet/pods/96f143d8-874e-4d12-94c5-51227b2d0740/volumes" Jul 14 18:23:26.679942 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:26.679899 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:26.680372 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:26.679992 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:26.684675 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:26.684651 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:26.914748 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:26.914715 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:23:26.973141 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:26.973045 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b755759b9-5hrkk"] Jul 14 18:23:27.780128 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:27.780036 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nlzhr" Jul 14 18:23:30.770621 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:30.770583 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:30.771104 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:30.770651 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:50.776006 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:50.775975 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:50.779801 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:50.779775 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-68ffd96d67-dmvng" Jul 14 18:23:52.000024 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:51.999982 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b755759b9-5hrkk" podUID="540d9122-3c8e-4099-8cf8-713d9b0833f7" containerName="console" containerID="cri-o://d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90" gracePeriod=15 Jul 14 18:23:52.232810 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.232783 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:52.334803 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.334760 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hglz9\" (UniqueName: \"kubernetes.io/projected/540d9122-3c8e-4099-8cf8-713d9b0833f7-kube-api-access-hglz9\") pod \"540d9122-3c8e-4099-8cf8-713d9b0833f7\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " Jul 14 18:23:52.334803 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.334799 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-service-ca\") pod \"540d9122-3c8e-4099-8cf8-713d9b0833f7\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " Jul 14 18:23:52.335080 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.334822 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-serving-cert\") pod \"540d9122-3c8e-4099-8cf8-713d9b0833f7\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " Jul 14 18:23:52.335080 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.334854 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-oauth-config\") pod \"540d9122-3c8e-4099-8cf8-713d9b0833f7\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " Jul 14 18:23:52.335080 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.335037 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-config\") pod \"540d9122-3c8e-4099-8cf8-713d9b0833f7\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " Jul 14 18:23:52.335236 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.335085 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-oauth-serving-cert\") pod \"540d9122-3c8e-4099-8cf8-713d9b0833f7\" (UID: \"540d9122-3c8e-4099-8cf8-713d9b0833f7\") " Jul 14 18:23:52.335432 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.335380 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-service-ca" (OuterVolumeSpecName: "service-ca") pod "540d9122-3c8e-4099-8cf8-713d9b0833f7" (UID: "540d9122-3c8e-4099-8cf8-713d9b0833f7"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:23:52.335432 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.335403 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-config" (OuterVolumeSpecName: "console-config") pod "540d9122-3c8e-4099-8cf8-713d9b0833f7" (UID: "540d9122-3c8e-4099-8cf8-713d9b0833f7"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:23:52.335565 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.335477 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "540d9122-3c8e-4099-8cf8-713d9b0833f7" (UID: "540d9122-3c8e-4099-8cf8-713d9b0833f7"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:23:52.337157 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.337124 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "540d9122-3c8e-4099-8cf8-713d9b0833f7" (UID: "540d9122-3c8e-4099-8cf8-713d9b0833f7"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:23:52.337303 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.337161 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/540d9122-3c8e-4099-8cf8-713d9b0833f7-kube-api-access-hglz9" (OuterVolumeSpecName: "kube-api-access-hglz9") pod "540d9122-3c8e-4099-8cf8-713d9b0833f7" (UID: "540d9122-3c8e-4099-8cf8-713d9b0833f7"). InnerVolumeSpecName "kube-api-access-hglz9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:23:52.337303 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.337164 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "540d9122-3c8e-4099-8cf8-713d9b0833f7" (UID: "540d9122-3c8e-4099-8cf8-713d9b0833f7"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:23:52.435752 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.435717 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-oauth-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:52.435752 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.435746 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:52.435752 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.435755 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-oauth-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:52.435752 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.435764 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hglz9\" (UniqueName: \"kubernetes.io/projected/540d9122-3c8e-4099-8cf8-713d9b0833f7-kube-api-access-hglz9\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:52.436006 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.435773 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/540d9122-3c8e-4099-8cf8-713d9b0833f7-service-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:52.436006 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:52.435782 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/540d9122-3c8e-4099-8cf8-713d9b0833f7-console-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:23:53.005782 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.005746 2569 generic.go:356] "Generic (PLEG): container finished" podID="540d9122-3c8e-4099-8cf8-713d9b0833f7" containerID="d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90" exitCode=0 Jul 14 18:23:53.006177 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.005804 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b755759b9-5hrkk" Jul 14 18:23:53.006177 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.005815 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b755759b9-5hrkk" event={"ID":"540d9122-3c8e-4099-8cf8-713d9b0833f7","Type":"ContainerDied","Data":"d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90"} Jul 14 18:23:53.006177 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.005897 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b755759b9-5hrkk" event={"ID":"540d9122-3c8e-4099-8cf8-713d9b0833f7","Type":"ContainerDied","Data":"90ddda2b9238d20f6285ea383f9fbf07432dd1686994f9292a6f8002bd9d0f0c"} Jul 14 18:23:53.006177 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.005925 2569 scope.go:117] "RemoveContainer" containerID="d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90" Jul 14 18:23:53.013982 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.013957 2569 scope.go:117] "RemoveContainer" containerID="d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90" Jul 14 18:23:53.014299 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:23:53.014282 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90\": container with ID starting with d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90 not found: ID does not exist" containerID="d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90" Jul 14 18:23:53.014361 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.014309 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90"} err="failed to get container status \"d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90\": rpc error: code = NotFound desc = could not find container \"d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90\": container with ID starting with d930e3e42cd5a8634f4e4e7d096a682bb3134bf15347e7d21afc9f2393beeb90 not found: ID does not exist" Jul 14 18:23:53.058779 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.058750 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b755759b9-5hrkk"] Jul 14 18:23:53.078991 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:53.078962 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b755759b9-5hrkk"] Jul 14 18:23:54.482663 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:23:54.482626 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="540d9122-3c8e-4099-8cf8-713d9b0833f7" path="/var/lib/kubelet/pods/540d9122-3c8e-4099-8cf8-713d9b0833f7/volumes" Jul 14 18:24:34.980248 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.980207 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66577c57dd-k9sbd"] Jul 14 18:24:34.980813 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.980491 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="540d9122-3c8e-4099-8cf8-713d9b0833f7" containerName="console" Jul 14 18:24:34.980813 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.980502 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="540d9122-3c8e-4099-8cf8-713d9b0833f7" containerName="console" Jul 14 18:24:34.980813 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.980513 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96f143d8-874e-4d12-94c5-51227b2d0740" containerName="registry" Jul 14 18:24:34.980813 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.980520 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="96f143d8-874e-4d12-94c5-51227b2d0740" containerName="registry" Jul 14 18:24:34.980813 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.980575 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="540d9122-3c8e-4099-8cf8-713d9b0833f7" containerName="console" Jul 14 18:24:34.980813 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.980584 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="96f143d8-874e-4d12-94c5-51227b2d0740" containerName="registry" Jul 14 18:24:34.982522 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.982499 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:34.997701 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:34.997668 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66577c57dd-k9sbd"] Jul 14 18:24:35.079480 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.079442 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-serving-cert\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.079668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.079490 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-oauth-config\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.079668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.079518 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-service-ca\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.079668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.079560 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-oauth-serving-cert\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.079668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.079609 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-console-config\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.079668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.079656 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw4k4\" (UniqueName: \"kubernetes.io/projected/b7060f86-274a-44f2-92d3-f395c910d5f2-kube-api-access-fw4k4\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.079858 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.079713 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-trusted-ca-bundle\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.180657 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.180617 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-oauth-serving-cert\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.180657 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.180657 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-console-config\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.180900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.180678 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw4k4\" (UniqueName: \"kubernetes.io/projected/b7060f86-274a-44f2-92d3-f395c910d5f2-kube-api-access-fw4k4\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.180900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.180788 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-trusted-ca-bundle\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.180900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.180855 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-serving-cert\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.180900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.180885 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-oauth-config\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.181103 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.180906 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-service-ca\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.181528 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.181489 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-oauth-serving-cert\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.181528 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.181516 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-console-config\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.181724 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.181594 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-service-ca\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.181724 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.181709 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-trusted-ca-bundle\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.183401 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.183368 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-oauth-config\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.183520 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.183500 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-serving-cert\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.191476 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.191450 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw4k4\" (UniqueName: \"kubernetes.io/projected/b7060f86-274a-44f2-92d3-f395c910d5f2-kube-api-access-fw4k4\") pod \"console-66577c57dd-k9sbd\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.293423 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.293386 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:35.424737 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:35.424702 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66577c57dd-k9sbd"] Jul 14 18:24:35.428562 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:24:35.428538 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7060f86_274a_44f2_92d3_f395c910d5f2.slice/crio-533b00a266814e8ca9ec2d82e2a58b871be0eda0641690f6362adf17d9ab4919 WatchSource:0}: Error finding container 533b00a266814e8ca9ec2d82e2a58b871be0eda0641690f6362adf17d9ab4919: Status 404 returned error can't find the container with id 533b00a266814e8ca9ec2d82e2a58b871be0eda0641690f6362adf17d9ab4919 Jul 14 18:24:36.158837 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:36.158803 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66577c57dd-k9sbd" event={"ID":"b7060f86-274a-44f2-92d3-f395c910d5f2","Type":"ContainerStarted","Data":"e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09"} Jul 14 18:24:36.158837 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:36.158856 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66577c57dd-k9sbd" event={"ID":"b7060f86-274a-44f2-92d3-f395c910d5f2","Type":"ContainerStarted","Data":"533b00a266814e8ca9ec2d82e2a58b871be0eda0641690f6362adf17d9ab4919"} Jul 14 18:24:36.180429 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:36.180382 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66577c57dd-k9sbd" podStartSLOduration=2.180365191 podStartE2EDuration="2.180365191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:24:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:24:36.179843232 +0000 UTC m=+170.237511301" watchObservedRunningTime="2026-07-14 18:24:36.180365191 +0000 UTC m=+170.238033241" Jul 14 18:24:45.294369 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:45.294241 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:45.294369 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:45.294328 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:45.299177 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:45.299155 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:46.195149 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:46.195121 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:24:46.259548 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:24:46.259509 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76ccf85775-dxvbp"] Jul 14 18:25:11.282500 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.282431 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-76ccf85775-dxvbp" podUID="052da12e-bea0-4906-aae4-fcc958cae59d" containerName="console" containerID="cri-o://e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407" gracePeriod=15 Jul 14 18:25:11.525210 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.525185 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:25:11.676163 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676071 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-oauth-config\") pod \"052da12e-bea0-4906-aae4-fcc958cae59d\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " Jul 14 18:25:11.676163 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676123 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-console-config\") pod \"052da12e-bea0-4906-aae4-fcc958cae59d\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " Jul 14 18:25:11.676163 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676151 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-service-ca\") pod \"052da12e-bea0-4906-aae4-fcc958cae59d\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " Jul 14 18:25:11.676163 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676166 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-serving-cert\") pod \"052da12e-bea0-4906-aae4-fcc958cae59d\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " Jul 14 18:25:11.676499 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676300 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-oauth-serving-cert\") pod \"052da12e-bea0-4906-aae4-fcc958cae59d\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " Jul 14 18:25:11.676499 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676344 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-trusted-ca-bundle\") pod \"052da12e-bea0-4906-aae4-fcc958cae59d\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " Jul 14 18:25:11.676499 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676404 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nnhmr\" (UniqueName: \"kubernetes.io/projected/052da12e-bea0-4906-aae4-fcc958cae59d-kube-api-access-nnhmr\") pod \"052da12e-bea0-4906-aae4-fcc958cae59d\" (UID: \"052da12e-bea0-4906-aae4-fcc958cae59d\") " Jul 14 18:25:11.676638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676539 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-console-config" (OuterVolumeSpecName: "console-config") pod "052da12e-bea0-4906-aae4-fcc958cae59d" (UID: "052da12e-bea0-4906-aae4-fcc958cae59d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:25:11.676638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676622 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-service-ca" (OuterVolumeSpecName: "service-ca") pod "052da12e-bea0-4906-aae4-fcc958cae59d" (UID: "052da12e-bea0-4906-aae4-fcc958cae59d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:25:11.676754 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676668 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "052da12e-bea0-4906-aae4-fcc958cae59d" (UID: "052da12e-bea0-4906-aae4-fcc958cae59d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:25:11.676811 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676792 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "052da12e-bea0-4906-aae4-fcc958cae59d" (UID: "052da12e-bea0-4906-aae4-fcc958cae59d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:25:11.676854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676821 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-console-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:25:11.676854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676839 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-service-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:25:11.676854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.676848 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-oauth-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:25:11.678545 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.678523 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/052da12e-bea0-4906-aae4-fcc958cae59d-kube-api-access-nnhmr" (OuterVolumeSpecName: "kube-api-access-nnhmr") pod "052da12e-bea0-4906-aae4-fcc958cae59d" (UID: "052da12e-bea0-4906-aae4-fcc958cae59d"). InnerVolumeSpecName "kube-api-access-nnhmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:25:11.678832 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.678811 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "052da12e-bea0-4906-aae4-fcc958cae59d" (UID: "052da12e-bea0-4906-aae4-fcc958cae59d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:25:11.678832 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.678824 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "052da12e-bea0-4906-aae4-fcc958cae59d" (UID: "052da12e-bea0-4906-aae4-fcc958cae59d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:25:11.778101 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.778064 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-oauth-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:25:11.778101 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.778095 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/052da12e-bea0-4906-aae4-fcc958cae59d-console-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:25:11.778101 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.778107 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/052da12e-bea0-4906-aae4-fcc958cae59d-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:25:11.778367 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:11.778116 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnhmr\" (UniqueName: \"kubernetes.io/projected/052da12e-bea0-4906-aae4-fcc958cae59d-kube-api-access-nnhmr\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:25:12.283120 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.283089 2569 generic.go:356] "Generic (PLEG): container finished" podID="052da12e-bea0-4906-aae4-fcc958cae59d" containerID="e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407" exitCode=0 Jul 14 18:25:12.283630 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.283156 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76ccf85775-dxvbp" event={"ID":"052da12e-bea0-4906-aae4-fcc958cae59d","Type":"ContainerDied","Data":"e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407"} Jul 14 18:25:12.283630 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.283161 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-76ccf85775-dxvbp" Jul 14 18:25:12.283630 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.283226 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-76ccf85775-dxvbp" event={"ID":"052da12e-bea0-4906-aae4-fcc958cae59d","Type":"ContainerDied","Data":"58fff18b1a5aebb0f3aac979cc6d1e47ca1604a50f1d696f61c9b4fcd9b31d38"} Jul 14 18:25:12.283630 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.283246 2569 scope.go:117] "RemoveContainer" containerID="e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407" Jul 14 18:25:12.291609 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.291588 2569 scope.go:117] "RemoveContainer" containerID="e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407" Jul 14 18:25:12.291882 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:25:12.291858 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407\": container with ID starting with e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407 not found: ID does not exist" containerID="e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407" Jul 14 18:25:12.291970 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.291890 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407"} err="failed to get container status \"e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407\": rpc error: code = NotFound desc = could not find container \"e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407\": container with ID starting with e9e10cfb1663888669e9e913c9f0cc9347f72e3fe4176f17c595c690d0b68407 not found: ID does not exist" Jul 14 18:25:12.306380 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.306358 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-76ccf85775-dxvbp"] Jul 14 18:25:12.310302 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.310281 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-76ccf85775-dxvbp"] Jul 14 18:25:12.480935 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:12.480902 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="052da12e-bea0-4906-aae4-fcc958cae59d" path="/var/lib/kubelet/pods/052da12e-bea0-4906-aae4-fcc958cae59d/volumes" Jul 14 18:25:55.034046 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.034012 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6844976579-cwnfs"] Jul 14 18:25:55.034457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.034248 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="052da12e-bea0-4906-aae4-fcc958cae59d" containerName="console" Jul 14 18:25:55.034457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.034275 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="052da12e-bea0-4906-aae4-fcc958cae59d" containerName="console" Jul 14 18:25:55.034457 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.034326 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="052da12e-bea0-4906-aae4-fcc958cae59d" containerName="console" Jul 14 18:25:55.036559 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.036544 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.103818 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.103781 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-serving-cert\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.104024 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.103825 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-trusted-ca-bundle\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.104024 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.103879 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-service-ca\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.104024 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.103974 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-oauth-serving-cert\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.104024 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.104008 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-oauth-config\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.104210 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.104029 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrrhf\" (UniqueName: \"kubernetes.io/projected/c50ad9c4-8104-4fd2-8518-822a65f98f73-kube-api-access-rrrhf\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.104210 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.104094 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-config\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.116336 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.116296 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6844976579-cwnfs"] Jul 14 18:25:55.204523 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.204480 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-serving-cert\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.204725 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.204541 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-trusted-ca-bundle\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.204725 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.204566 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-service-ca\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.204725 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.204593 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-oauth-serving-cert\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.204725 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.204622 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-oauth-config\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.204725 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.204640 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrrhf\" (UniqueName: \"kubernetes.io/projected/c50ad9c4-8104-4fd2-8518-822a65f98f73-kube-api-access-rrrhf\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.204725 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.204674 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-config\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.205407 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.205384 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-config\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.205544 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.205449 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-oauth-serving-cert\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.205544 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.205502 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-service-ca\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.205801 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.205784 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-trusted-ca-bundle\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.207539 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.207521 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-oauth-config\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.207743 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.207722 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-serving-cert\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.216843 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.216815 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrrhf\" (UniqueName: \"kubernetes.io/projected/c50ad9c4-8104-4fd2-8518-822a65f98f73-kube-api-access-rrrhf\") pod \"console-6844976579-cwnfs\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.345291 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.345178 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:25:55.492345 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:55.492313 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6844976579-cwnfs"] Jul 14 18:25:55.495655 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:25:55.495625 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc50ad9c4_8104_4fd2_8518_822a65f98f73.slice/crio-08736449ce4f21bbfa35cebbfb1c10f3fc2df19a67eecfb6a31267f924a34a76 WatchSource:0}: Error finding container 08736449ce4f21bbfa35cebbfb1c10f3fc2df19a67eecfb6a31267f924a34a76: Status 404 returned error can't find the container with id 08736449ce4f21bbfa35cebbfb1c10f3fc2df19a67eecfb6a31267f924a34a76 Jul 14 18:25:56.424583 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:56.424548 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6844976579-cwnfs" event={"ID":"c50ad9c4-8104-4fd2-8518-822a65f98f73","Type":"ContainerStarted","Data":"3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1"} Jul 14 18:25:56.424976 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:25:56.424606 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6844976579-cwnfs" event={"ID":"c50ad9c4-8104-4fd2-8518-822a65f98f73","Type":"ContainerStarted","Data":"08736449ce4f21bbfa35cebbfb1c10f3fc2df19a67eecfb6a31267f924a34a76"} Jul 14 18:26:05.345879 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:05.345831 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:26:05.346376 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:05.345905 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:26:05.350348 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:05.350328 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:26:05.388287 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:05.388222 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6844976579-cwnfs" podStartSLOduration=11.388207547 podStartE2EDuration="11.388207547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:25:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:25:56.459838649 +0000 UTC m=+250.517506699" watchObservedRunningTime="2026-07-14 18:26:05.388207547 +0000 UTC m=+259.445875594" Jul 14 18:26:05.457132 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:05.457104 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:26:05.549970 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:05.549939 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66577c57dd-k9sbd"] Jul 14 18:26:30.568710 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.568646 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66577c57dd-k9sbd" podUID="b7060f86-274a-44f2-92d3-f395c910d5f2" containerName="console" containerID="cri-o://e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09" gracePeriod=15 Jul 14 18:26:30.800971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.800947 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:26:30.989356 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989230 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-service-ca\") pod \"b7060f86-274a-44f2-92d3-f395c910d5f2\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " Jul 14 18:26:30.989356 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989296 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-console-config\") pod \"b7060f86-274a-44f2-92d3-f395c910d5f2\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " Jul 14 18:26:30.989356 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989342 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-trusted-ca-bundle\") pod \"b7060f86-274a-44f2-92d3-f395c910d5f2\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " Jul 14 18:26:30.989634 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989396 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-oauth-serving-cert\") pod \"b7060f86-274a-44f2-92d3-f395c910d5f2\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " Jul 14 18:26:30.989634 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989426 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-serving-cert\") pod \"b7060f86-274a-44f2-92d3-f395c910d5f2\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " Jul 14 18:26:30.989634 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989458 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw4k4\" (UniqueName: \"kubernetes.io/projected/b7060f86-274a-44f2-92d3-f395c910d5f2-kube-api-access-fw4k4\") pod \"b7060f86-274a-44f2-92d3-f395c910d5f2\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " Jul 14 18:26:30.989634 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989527 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-oauth-config\") pod \"b7060f86-274a-44f2-92d3-f395c910d5f2\" (UID: \"b7060f86-274a-44f2-92d3-f395c910d5f2\") " Jul 14 18:26:30.989824 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989785 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-console-config" (OuterVolumeSpecName: "console-config") pod "b7060f86-274a-44f2-92d3-f395c910d5f2" (UID: "b7060f86-274a-44f2-92d3-f395c910d5f2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:26:30.989824 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989798 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-service-ca" (OuterVolumeSpecName: "service-ca") pod "b7060f86-274a-44f2-92d3-f395c910d5f2" (UID: "b7060f86-274a-44f2-92d3-f395c910d5f2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:26:30.989918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.989824 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "b7060f86-274a-44f2-92d3-f395c910d5f2" (UID: "b7060f86-274a-44f2-92d3-f395c910d5f2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:26:30.990127 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.990099 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "b7060f86-274a-44f2-92d3-f395c910d5f2" (UID: "b7060f86-274a-44f2-92d3-f395c910d5f2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:26:30.991651 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.991624 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "b7060f86-274a-44f2-92d3-f395c910d5f2" (UID: "b7060f86-274a-44f2-92d3-f395c910d5f2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:26:30.991747 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.991658 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7060f86-274a-44f2-92d3-f395c910d5f2-kube-api-access-fw4k4" (OuterVolumeSpecName: "kube-api-access-fw4k4") pod "b7060f86-274a-44f2-92d3-f395c910d5f2" (UID: "b7060f86-274a-44f2-92d3-f395c910d5f2"). InnerVolumeSpecName "kube-api-access-fw4k4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:26:30.991786 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:30.991762 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "b7060f86-274a-44f2-92d3-f395c910d5f2" (UID: "b7060f86-274a-44f2-92d3-f395c910d5f2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:26:31.090340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.090302 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-service-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:31.090340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.090333 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-console-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:31.090340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.090345 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:31.090579 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.090357 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b7060f86-274a-44f2-92d3-f395c910d5f2-oauth-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:31.090579 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.090369 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:31.090579 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.090379 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw4k4\" (UniqueName: \"kubernetes.io/projected/b7060f86-274a-44f2-92d3-f395c910d5f2-kube-api-access-fw4k4\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:31.090579 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.090393 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b7060f86-274a-44f2-92d3-f395c910d5f2-console-oauth-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:31.538359 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.538330 2569 generic.go:356] "Generic (PLEG): container finished" podID="b7060f86-274a-44f2-92d3-f395c910d5f2" containerID="e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09" exitCode=0 Jul 14 18:26:31.538535 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.538375 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66577c57dd-k9sbd" Jul 14 18:26:31.538535 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.538393 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66577c57dd-k9sbd" event={"ID":"b7060f86-274a-44f2-92d3-f395c910d5f2","Type":"ContainerDied","Data":"e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09"} Jul 14 18:26:31.538535 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.538430 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66577c57dd-k9sbd" event={"ID":"b7060f86-274a-44f2-92d3-f395c910d5f2","Type":"ContainerDied","Data":"533b00a266814e8ca9ec2d82e2a58b871be0eda0641690f6362adf17d9ab4919"} Jul 14 18:26:31.538535 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.538449 2569 scope.go:117] "RemoveContainer" containerID="e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09" Jul 14 18:26:31.546420 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.546404 2569 scope.go:117] "RemoveContainer" containerID="e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09" Jul 14 18:26:31.546677 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:26:31.546657 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09\": container with ID starting with e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09 not found: ID does not exist" containerID="e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09" Jul 14 18:26:31.546801 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.546684 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09"} err="failed to get container status \"e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09\": rpc error: code = NotFound desc = could not find container \"e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09\": container with ID starting with e16452845f28b5451ce294e41667b28cfeb276ed93eee2f48f354690d11eed09 not found: ID does not exist" Jul 14 18:26:31.563713 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.563685 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66577c57dd-k9sbd"] Jul 14 18:26:31.573654 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:31.573627 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66577c57dd-k9sbd"] Jul 14 18:26:32.480493 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:32.480460 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7060f86-274a-44f2-92d3-f395c910d5f2" path="/var/lib/kubelet/pods/b7060f86-274a-44f2-92d3-f395c910d5f2/volumes" Jul 14 18:26:35.408160 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.408113 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8"] Jul 14 18:26:35.408637 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.408404 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7060f86-274a-44f2-92d3-f395c910d5f2" containerName="console" Jul 14 18:26:35.408637 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.408415 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7060f86-274a-44f2-92d3-f395c910d5f2" containerName="console" Jul 14 18:26:35.408637 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.408458 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7060f86-274a-44f2-92d3-f395c910d5f2" containerName="console" Jul 14 18:26:35.411661 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.411645 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.416744 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.416721 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 18:26:35.416871 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.416824 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pv6jk\"" Jul 14 18:26:35.417888 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.417868 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 18:26:35.420566 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.420546 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.420661 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.420588 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.420661 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.420618 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bcpj9\" (UniqueName: \"kubernetes.io/projected/d641f1b9-6fe7-493d-9f35-4f5400538387-kube-api-access-bcpj9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.424622 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.424598 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8"] Jul 14 18:26:35.521908 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.521872 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.522091 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.521923 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.522091 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.521941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bcpj9\" (UniqueName: \"kubernetes.io/projected/d641f1b9-6fe7-493d-9f35-4f5400538387-kube-api-access-bcpj9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.522358 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.522339 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.522407 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.522381 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.535705 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.535670 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bcpj9\" (UniqueName: \"kubernetes.io/projected/d641f1b9-6fe7-493d-9f35-4f5400538387-kube-api-access-bcpj9\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.720155 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.720066 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:35.851724 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:35.851690 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8"] Jul 14 18:26:35.854570 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:26:35.854538 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd641f1b9_6fe7_493d_9f35_4f5400538387.slice/crio-c255e833d9bbfe3cad4894d2ce92259fef31196df00a1a1a8f23a13959dfaf84 WatchSource:0}: Error finding container c255e833d9bbfe3cad4894d2ce92259fef31196df00a1a1a8f23a13959dfaf84: Status 404 returned error can't find the container with id c255e833d9bbfe3cad4894d2ce92259fef31196df00a1a1a8f23a13959dfaf84 Jul 14 18:26:36.560303 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:36.560267 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" event={"ID":"d641f1b9-6fe7-493d-9f35-4f5400538387","Type":"ContainerStarted","Data":"c255e833d9bbfe3cad4894d2ce92259fef31196df00a1a1a8f23a13959dfaf84"} Jul 14 18:26:43.588602 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:43.588568 2569 generic.go:356] "Generic (PLEG): container finished" podID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerID="bac747c4efa3e6eaadb2ea3ac8186ddc77d31714ea6a7b1410aa41cd4bf245e1" exitCode=0 Jul 14 18:26:43.588998 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:43.588617 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" event={"ID":"d641f1b9-6fe7-493d-9f35-4f5400538387","Type":"ContainerDied","Data":"bac747c4efa3e6eaadb2ea3ac8186ddc77d31714ea6a7b1410aa41cd4bf245e1"} Jul 14 18:26:46.930239 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:46.930216 2569 kubelet.go:1648] "Image garbage collection succeeded" Jul 14 18:26:47.604367 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:47.604337 2569 generic.go:356] "Generic (PLEG): container finished" podID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerID="aa27867c61000436684c8ef573521a77a19a188ef08064036458c3f749de1608" exitCode=0 Jul 14 18:26:47.604531 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:47.604426 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" event={"ID":"d641f1b9-6fe7-493d-9f35-4f5400538387","Type":"ContainerDied","Data":"aa27867c61000436684c8ef573521a77a19a188ef08064036458c3f749de1608"} Jul 14 18:26:47.605390 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:47.605374 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:26:56.638135 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:56.638099 2569 generic.go:356] "Generic (PLEG): container finished" podID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerID="5179362946a9cbd835dcfd1726211d98cba6e061b451d7f9e98a5c9a24e93327" exitCode=0 Jul 14 18:26:56.638510 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:56.638173 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" event={"ID":"d641f1b9-6fe7-493d-9f35-4f5400538387","Type":"ContainerDied","Data":"5179362946a9cbd835dcfd1726211d98cba6e061b451d7f9e98a5c9a24e93327"} Jul 14 18:26:57.803821 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:57.803787 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:57.900086 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:57.899994 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-bundle\") pod \"d641f1b9-6fe7-493d-9f35-4f5400538387\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " Jul 14 18:26:57.900086 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:57.900054 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bcpj9\" (UniqueName: \"kubernetes.io/projected/d641f1b9-6fe7-493d-9f35-4f5400538387-kube-api-access-bcpj9\") pod \"d641f1b9-6fe7-493d-9f35-4f5400538387\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " Jul 14 18:26:57.900086 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:57.900087 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-util\") pod \"d641f1b9-6fe7-493d-9f35-4f5400538387\" (UID: \"d641f1b9-6fe7-493d-9f35-4f5400538387\") " Jul 14 18:26:57.900784 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:57.900753 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-bundle" (OuterVolumeSpecName: "bundle") pod "d641f1b9-6fe7-493d-9f35-4f5400538387" (UID: "d641f1b9-6fe7-493d-9f35-4f5400538387"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:26:57.902270 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:57.902230 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d641f1b9-6fe7-493d-9f35-4f5400538387-kube-api-access-bcpj9" (OuterVolumeSpecName: "kube-api-access-bcpj9") pod "d641f1b9-6fe7-493d-9f35-4f5400538387" (UID: "d641f1b9-6fe7-493d-9f35-4f5400538387"). InnerVolumeSpecName "kube-api-access-bcpj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:26:57.904757 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:57.904732 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-util" (OuterVolumeSpecName: "util") pod "d641f1b9-6fe7-493d-9f35-4f5400538387" (UID: "d641f1b9-6fe7-493d-9f35-4f5400538387"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:26:58.000582 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:58.000544 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-util\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:58.000582 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:58.000576 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d641f1b9-6fe7-493d-9f35-4f5400538387-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:58.000582 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:58.000585 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bcpj9\" (UniqueName: \"kubernetes.io/projected/d641f1b9-6fe7-493d-9f35-4f5400538387-kube-api-access-bcpj9\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:26:58.651269 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:58.651227 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" event={"ID":"d641f1b9-6fe7-493d-9f35-4f5400538387","Type":"ContainerDied","Data":"c255e833d9bbfe3cad4894d2ce92259fef31196df00a1a1a8f23a13959dfaf84"} Jul 14 18:26:58.651269 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:58.651244 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540ktzf8" Jul 14 18:26:58.651482 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:26:58.651293 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c255e833d9bbfe3cad4894d2ce92259fef31196df00a1a1a8f23a13959dfaf84" Jul 14 18:27:03.881101 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.881071 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2"] Jul 14 18:27:03.881516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.881332 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerName="pull" Jul 14 18:27:03.881516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.881343 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerName="pull" Jul 14 18:27:03.881516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.881352 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerName="util" Jul 14 18:27:03.881516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.881358 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerName="util" Jul 14 18:27:03.881516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.881364 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerName="extract" Jul 14 18:27:03.881516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.881369 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerName="extract" Jul 14 18:27:03.881516 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.881409 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="d641f1b9-6fe7-493d-9f35-4f5400538387" containerName="extract" Jul 14 18:27:03.884172 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.884156 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" Jul 14 18:27:03.887131 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.887108 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:27:03.887791 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.887773 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 14 18:27:03.887889 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.887874 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-p2tbv\"" Jul 14 18:27:03.902666 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.902638 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2"] Jul 14 18:27:03.947043 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.947010 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3808ccc-aa91-436e-bced-a4c6619dacaa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jfp2\" (UID: \"b3808ccc-aa91-436e-bced-a4c6619dacaa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" Jul 14 18:27:03.947043 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:03.947054 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6jqm\" (UniqueName: \"kubernetes.io/projected/b3808ccc-aa91-436e-bced-a4c6619dacaa-kube-api-access-m6jqm\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jfp2\" (UID: \"b3808ccc-aa91-436e-bced-a4c6619dacaa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" Jul 14 18:27:04.048080 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:04.048044 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3808ccc-aa91-436e-bced-a4c6619dacaa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jfp2\" (UID: \"b3808ccc-aa91-436e-bced-a4c6619dacaa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" Jul 14 18:27:04.048234 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:04.048091 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6jqm\" (UniqueName: \"kubernetes.io/projected/b3808ccc-aa91-436e-bced-a4c6619dacaa-kube-api-access-m6jqm\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jfp2\" (UID: \"b3808ccc-aa91-436e-bced-a4c6619dacaa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" Jul 14 18:27:04.048495 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:04.048474 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b3808ccc-aa91-436e-bced-a4c6619dacaa-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jfp2\" (UID: \"b3808ccc-aa91-436e-bced-a4c6619dacaa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" Jul 14 18:27:04.058721 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:04.058688 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6jqm\" (UniqueName: \"kubernetes.io/projected/b3808ccc-aa91-436e-bced-a4c6619dacaa-kube-api-access-m6jqm\") pod \"cert-manager-operator-controller-manager-55c444bf86-9jfp2\" (UID: \"b3808ccc-aa91-436e-bced-a4c6619dacaa\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" Jul 14 18:27:04.193618 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:04.193521 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" Jul 14 18:27:04.343892 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:04.343852 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2"] Jul 14 18:27:04.346363 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:27:04.346334 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3808ccc_aa91_436e_bced_a4c6619dacaa.slice/crio-860e9e7e12eb5959a574261240eab09ca394349956785e5f9d6c13e2fee2a922 WatchSource:0}: Error finding container 860e9e7e12eb5959a574261240eab09ca394349956785e5f9d6c13e2fee2a922: Status 404 returned error can't find the container with id 860e9e7e12eb5959a574261240eab09ca394349956785e5f9d6c13e2fee2a922 Jul 14 18:27:04.672936 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:04.672900 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" event={"ID":"b3808ccc-aa91-436e-bced-a4c6619dacaa","Type":"ContainerStarted","Data":"860e9e7e12eb5959a574261240eab09ca394349956785e5f9d6c13e2fee2a922"} Jul 14 18:27:06.682835 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:06.682741 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" event={"ID":"b3808ccc-aa91-436e-bced-a4c6619dacaa","Type":"ContainerStarted","Data":"5206f2e54ed0d93de89247d33f3df6fde11336f90415ac87d6c980206b7c4900"} Jul 14 18:27:07.405095 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.405043 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-9jfp2" podStartSLOduration=2.490957342 podStartE2EDuration="4.405023905s" totalImagesPullingTime="1.914066563s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:06.723152578 +0000 UTC m=+320.780820627" watchObservedRunningTime="2026-07-14 18:27:07.405023905 +0000 UTC m=+321.462691954" Jul 14 18:27:07.405321 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.405304 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t"] Jul 14 18:27:07.407336 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.407321 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.413740 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.413718 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 18:27:07.413838 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.413763 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 18:27:07.414693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.414674 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pv6jk\"" Jul 14 18:27:07.431076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.431050 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t"] Jul 14 18:27:07.476861 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.476826 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.476861 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.476867 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.477098 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.476888 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr88b\" (UniqueName: \"kubernetes.io/projected/1d768dd3-b8c2-4189-930d-af13edbc9d04-kube-api-access-wr88b\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.577441 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.577400 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.577441 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.577442 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.577702 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.577466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr88b\" (UniqueName: \"kubernetes.io/projected/1d768dd3-b8c2-4189-930d-af13edbc9d04-kube-api-access-wr88b\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.577868 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.577844 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.577937 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.577891 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.596874 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.596842 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr88b\" (UniqueName: \"kubernetes.io/projected/1d768dd3-b8c2-4189-930d-af13edbc9d04-kube-api-access-wr88b\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.715804 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.715707 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:07.890245 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:07.890213 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t"] Jul 14 18:27:07.895444 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:27:07.895419 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d768dd3_b8c2_4189_930d_af13edbc9d04.slice/crio-dad88301dbc9be0d313375791b25a90f46f9fbd27122b07fb7d4cf42c3c1bd0f WatchSource:0}: Error finding container dad88301dbc9be0d313375791b25a90f46f9fbd27122b07fb7d4cf42c3c1bd0f: Status 404 returned error can't find the container with id dad88301dbc9be0d313375791b25a90f46f9fbd27122b07fb7d4cf42c3c1bd0f Jul 14 18:27:08.692388 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:08.692354 2569 generic.go:356] "Generic (PLEG): container finished" podID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerID="a8eafe4aa9ab9cc7c14ef4b0f43fc6b4714f3b2e94afedeedf4ee7854fc88d0b" exitCode=0 Jul 14 18:27:08.692572 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:08.692405 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" event={"ID":"1d768dd3-b8c2-4189-930d-af13edbc9d04","Type":"ContainerDied","Data":"a8eafe4aa9ab9cc7c14ef4b0f43fc6b4714f3b2e94afedeedf4ee7854fc88d0b"} Jul 14 18:27:08.692572 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:08.692442 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" event={"ID":"1d768dd3-b8c2-4189-930d-af13edbc9d04","Type":"ContainerStarted","Data":"dad88301dbc9be0d313375791b25a90f46f9fbd27122b07fb7d4cf42c3c1bd0f"} Jul 14 18:27:11.705761 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:11.705731 2569 generic.go:356] "Generic (PLEG): container finished" podID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerID="bbe86b91c0af7723dbaf452c03c9b5c284dbcf550ac7f7543512185d87beea44" exitCode=0 Jul 14 18:27:11.706206 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:11.705770 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" event={"ID":"1d768dd3-b8c2-4189-930d-af13edbc9d04","Type":"ContainerDied","Data":"bbe86b91c0af7723dbaf452c03c9b5c284dbcf550ac7f7543512185d87beea44"} Jul 14 18:27:12.711177 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:12.711146 2569 generic.go:356] "Generic (PLEG): container finished" podID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerID="ee413cebecae2b5bd6ea0fbe69f2a64f20ed8a1630fcb18c37e77d20c5a95cd2" exitCode=0 Jul 14 18:27:12.711573 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:12.711186 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" event={"ID":"1d768dd3-b8c2-4189-930d-af13edbc9d04","Type":"ContainerDied","Data":"ee413cebecae2b5bd6ea0fbe69f2a64f20ed8a1630fcb18c37e77d20c5a95cd2"} Jul 14 18:27:13.838453 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:13.838424 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:13.933513 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:13.933475 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr88b\" (UniqueName: \"kubernetes.io/projected/1d768dd3-b8c2-4189-930d-af13edbc9d04-kube-api-access-wr88b\") pod \"1d768dd3-b8c2-4189-930d-af13edbc9d04\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " Jul 14 18:27:13.933674 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:13.933526 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-util\") pod \"1d768dd3-b8c2-4189-930d-af13edbc9d04\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " Jul 14 18:27:13.933674 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:13.933550 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-bundle\") pod \"1d768dd3-b8c2-4189-930d-af13edbc9d04\" (UID: \"1d768dd3-b8c2-4189-930d-af13edbc9d04\") " Jul 14 18:27:13.933961 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:13.933938 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-bundle" (OuterVolumeSpecName: "bundle") pod "1d768dd3-b8c2-4189-930d-af13edbc9d04" (UID: "1d768dd3-b8c2-4189-930d-af13edbc9d04"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:27:13.935770 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:13.935742 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d768dd3-b8c2-4189-930d-af13edbc9d04-kube-api-access-wr88b" (OuterVolumeSpecName: "kube-api-access-wr88b") pod "1d768dd3-b8c2-4189-930d-af13edbc9d04" (UID: "1d768dd3-b8c2-4189-930d-af13edbc9d04"). InnerVolumeSpecName "kube-api-access-wr88b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:27:13.938304 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:13.938282 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-util" (OuterVolumeSpecName: "util") pod "1d768dd3-b8c2-4189-930d-af13edbc9d04" (UID: "1d768dd3-b8c2-4189-930d-af13edbc9d04"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:27:14.034291 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:14.034235 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr88b\" (UniqueName: \"kubernetes.io/projected/1d768dd3-b8c2-4189-930d-af13edbc9d04-kube-api-access-wr88b\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:27:14.034291 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:14.034289 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-util\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:27:14.034481 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:14.034300 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/1d768dd3-b8c2-4189-930d-af13edbc9d04-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:27:14.721310 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:14.721278 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" event={"ID":"1d768dd3-b8c2-4189-930d-af13edbc9d04","Type":"ContainerDied","Data":"dad88301dbc9be0d313375791b25a90f46f9fbd27122b07fb7d4cf42c3c1bd0f"} Jul 14 18:27:14.721310 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:14.721307 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fg685t" Jul 14 18:27:14.721506 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:14.721326 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dad88301dbc9be0d313375791b25a90f46f9fbd27122b07fb7d4cf42c3c1bd0f" Jul 14 18:27:21.210277 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.210224 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw"] Jul 14 18:27:21.210641 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.210490 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerName="pull" Jul 14 18:27:21.210641 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.210502 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerName="pull" Jul 14 18:27:21.210641 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.210512 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerName="util" Jul 14 18:27:21.210641 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.210516 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerName="util" Jul 14 18:27:21.210641 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.210523 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerName="extract" Jul 14 18:27:21.210641 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.210530 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerName="extract" Jul 14 18:27:21.210641 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.210574 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d768dd3-b8c2-4189-930d-af13edbc9d04" containerName="extract" Jul 14 18:27:21.212389 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.212373 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" Jul 14 18:27:21.221128 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.221105 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 14 18:27:21.222505 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.222488 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-jnzx7\"" Jul 14 18:27:21.222552 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.222492 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 14 18:27:21.253423 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.253393 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw"] Jul 14 18:27:21.297481 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.297445 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ncqg\" (UniqueName: \"kubernetes.io/projected/5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8-kube-api-access-5ncqg\") pod \"openshift-lws-operator-fd8ccff4c-j5gcw\" (UID: \"5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" Jul 14 18:27:21.297727 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.297511 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j5gcw\" (UID: \"5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" Jul 14 18:27:21.398241 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.398201 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ncqg\" (UniqueName: \"kubernetes.io/projected/5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8-kube-api-access-5ncqg\") pod \"openshift-lws-operator-fd8ccff4c-j5gcw\" (UID: \"5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" Jul 14 18:27:21.398434 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.398315 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j5gcw\" (UID: \"5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" Jul 14 18:27:21.398666 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.398649 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8-tmp\") pod \"openshift-lws-operator-fd8ccff4c-j5gcw\" (UID: \"5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" Jul 14 18:27:21.412011 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.411985 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ncqg\" (UniqueName: \"kubernetes.io/projected/5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8-kube-api-access-5ncqg\") pod \"openshift-lws-operator-fd8ccff4c-j5gcw\" (UID: \"5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" Jul 14 18:27:21.520905 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.520872 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" Jul 14 18:27:21.662642 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.662618 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw"] Jul 14 18:27:21.663814 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:27:21.663780 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5ad1b22b_f53a_4ee5_bb47_0a6197a0cbd8.slice/crio-875703dbdada4591da1169d82f041695a5f3f413bbfa98bef65570a7eba79886 WatchSource:0}: Error finding container 875703dbdada4591da1169d82f041695a5f3f413bbfa98bef65570a7eba79886: Status 404 returned error can't find the container with id 875703dbdada4591da1169d82f041695a5f3f413bbfa98bef65570a7eba79886 Jul 14 18:27:21.754593 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:21.754555 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" event={"ID":"5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8","Type":"ContainerStarted","Data":"875703dbdada4591da1169d82f041695a5f3f413bbfa98bef65570a7eba79886"} Jul 14 18:27:23.764712 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:23.764680 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" event={"ID":"5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8","Type":"ContainerStarted","Data":"7d1bfa87fc3a8659031c715b24b79f1a5e39c55d5ba022137d2a3267ecad2e90"} Jul 14 18:27:23.787963 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:23.787912 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-j5gcw" podStartSLOduration=1.184481798 podStartE2EDuration="2.787894291s" totalImagesPullingTime="1.603412493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:23.787407063 +0000 UTC m=+337.845075113" watchObservedRunningTime="2026-07-14 18:27:23.787894291 +0000 UTC m=+337.845562340" Jul 14 18:27:26.632472 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.632443 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2"] Jul 14 18:27:26.655509 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.655465 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2"] Jul 14 18:27:26.655696 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.655569 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.659311 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.659279 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-pv6jk\"" Jul 14 18:27:26.660327 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.660306 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 14 18:27:26.660892 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.660871 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 14 18:27:26.844565 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.844528 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.844565 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.844568 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.844775 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.844660 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brqg7\" (UniqueName: \"kubernetes.io/projected/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-kube-api-access-brqg7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.945652 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.945563 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.945652 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.945601 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.945850 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.945676 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brqg7\" (UniqueName: \"kubernetes.io/projected/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-kube-api-access-brqg7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.945954 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.945936 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.946035 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.946016 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.956830 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.956807 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brqg7\" (UniqueName: \"kubernetes.io/projected/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-kube-api-access-brqg7\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:26.964659 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:26.964634 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:27.131034 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:27.130986 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2"] Jul 14 18:27:27.133232 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:27:27.133198 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ec91d5e_c6ca_4ccd_80dd_02e5c6a95975.slice/crio-05adc5de4b30a9a97c951eec06ee7aef5ac175e5fbdc3de6eadb94b9e01cd04c WatchSource:0}: Error finding container 05adc5de4b30a9a97c951eec06ee7aef5ac175e5fbdc3de6eadb94b9e01cd04c: Status 404 returned error can't find the container with id 05adc5de4b30a9a97c951eec06ee7aef5ac175e5fbdc3de6eadb94b9e01cd04c Jul 14 18:27:27.783519 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:27.783432 2569 generic.go:356] "Generic (PLEG): container finished" podID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerID="9c44178fcf0cc54ff376eabb1d8c9cbf7eb7e5925263516c3faf0421c904e43d" exitCode=0 Jul 14 18:27:27.783519 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:27.783472 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" event={"ID":"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975","Type":"ContainerDied","Data":"9c44178fcf0cc54ff376eabb1d8c9cbf7eb7e5925263516c3faf0421c904e43d"} Jul 14 18:27:27.783519 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:27.783513 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" event={"ID":"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975","Type":"ContainerStarted","Data":"05adc5de4b30a9a97c951eec06ee7aef5ac175e5fbdc3de6eadb94b9e01cd04c"} Jul 14 18:27:29.793080 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:29.793050 2569 generic.go:356] "Generic (PLEG): container finished" podID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerID="c5f2703e27c17596f5cb5ae7003b258fbbfcd7ba50825c0bafc39a162fcd7b42" exitCode=0 Jul 14 18:27:29.793529 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:29.793098 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" event={"ID":"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975","Type":"ContainerDied","Data":"c5f2703e27c17596f5cb5ae7003b258fbbfcd7ba50825c0bafc39a162fcd7b42"} Jul 14 18:27:30.799649 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:30.799605 2569 generic.go:356] "Generic (PLEG): container finished" podID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerID="358f924d6faf7c59a8e3fa59c5e6a009c52e47702dd1a2c940a1b7ef90fbaab8" exitCode=0 Jul 14 18:27:30.800028 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:30.799658 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" event={"ID":"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975","Type":"ContainerDied","Data":"358f924d6faf7c59a8e3fa59c5e6a009c52e47702dd1a2c940a1b7ef90fbaab8"} Jul 14 18:27:31.941459 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:31.941431 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:31.984863 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:31.984828 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-util\") pod \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " Jul 14 18:27:31.985060 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:31.984887 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-bundle\") pod \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " Jul 14 18:27:31.985060 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:31.984936 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-brqg7\" (UniqueName: \"kubernetes.io/projected/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-kube-api-access-brqg7\") pod \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\" (UID: \"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975\") " Jul 14 18:27:31.985645 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:31.985611 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-bundle" (OuterVolumeSpecName: "bundle") pod "3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" (UID: "3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:27:31.987188 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:31.987151 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-kube-api-access-brqg7" (OuterVolumeSpecName: "kube-api-access-brqg7") pod "3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" (UID: "3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975"). InnerVolumeSpecName "kube-api-access-brqg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:27:31.990102 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:31.990078 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-util" (OuterVolumeSpecName: "util") pod "3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" (UID: "3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:27:32.085710 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:32.085624 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-util\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:27:32.085710 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:32.085655 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:27:32.085710 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:32.085664 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brqg7\" (UniqueName: \"kubernetes.io/projected/3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975-kube-api-access-brqg7\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:27:32.809362 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:32.809328 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" event={"ID":"3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975","Type":"ContainerDied","Data":"05adc5de4b30a9a97c951eec06ee7aef5ac175e5fbdc3de6eadb94b9e01cd04c"} Jul 14 18:27:32.809523 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:32.809371 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afntqp2" Jul 14 18:27:32.809523 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:32.809378 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="05adc5de4b30a9a97c951eec06ee7aef5ac175e5fbdc3de6eadb94b9e01cd04c" Jul 14 18:27:40.531559 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.531472 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f"] Jul 14 18:27:40.532108 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.531820 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerName="extract" Jul 14 18:27:40.532108 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.531837 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerName="extract" Jul 14 18:27:40.532108 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.531853 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerName="util" Jul 14 18:27:40.532108 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.531860 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerName="util" Jul 14 18:27:40.532108 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.531891 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerName="pull" Jul 14 18:27:40.532108 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.531901 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerName="pull" Jul 14 18:27:40.532108 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.531960 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ec91d5e-c6ca-4ccd-80dd-02e5c6a95975" containerName="extract" Jul 14 18:27:40.535723 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.535702 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.546360 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.546341 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-h5s8h\"" Jul 14 18:27:40.546503 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.546488 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 14 18:27:40.547647 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.547629 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.547749 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.547657 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlblj\" (UniqueName: \"kubernetes.io/projected/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-kube-api-access-jlblj\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.547749 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.547677 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.547846 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.547748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.547846 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.547783 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.555392 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.555372 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 14 18:27:40.555522 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.555461 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 14 18:27:40.555522 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.555513 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 14 18:27:40.576986 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.576960 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f"] Jul 14 18:27:40.648480 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.648442 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.648668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.648493 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.648668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.648523 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.648668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.648557 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.648668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.648578 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlblj\" (UniqueName: \"kubernetes.io/projected/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-kube-api-access-jlblj\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.648951 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.648915 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-manifests\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.651100 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.651065 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-tmp\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.651209 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.651139 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-webhook-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.651272 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.651218 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.770372 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.770342 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlblj\" (UniqueName: \"kubernetes.io/projected/0ed3bfaf-da25-4d88-a142-a15ee0000cf1-kube-api-access-jlblj\") pod \"opendatahub-operator-controller-manager-5d948cf9bc-c9r9f\" (UID: \"0ed3bfaf-da25-4d88-a142-a15ee0000cf1\") " pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:40.845412 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:40.845326 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:41.072657 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:41.072628 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f"] Jul 14 18:27:41.074455 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:27:41.074434 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed3bfaf_da25_4d88_a142_a15ee0000cf1.slice/crio-525d7ddc79d0b3744761af3a80c569c4b94fb2776c60604c692da20daa285fa7 WatchSource:0}: Error finding container 525d7ddc79d0b3744761af3a80c569c4b94fb2776c60604c692da20daa285fa7: Status 404 returned error can't find the container with id 525d7ddc79d0b3744761af3a80c569c4b94fb2776c60604c692da20daa285fa7 Jul 14 18:27:41.847656 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:41.847610 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" event={"ID":"0ed3bfaf-da25-4d88-a142-a15ee0000cf1","Type":"ContainerStarted","Data":"525d7ddc79d0b3744761af3a80c569c4b94fb2776c60604c692da20daa285fa7"} Jul 14 18:27:44.862990 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:44.862953 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" event={"ID":"0ed3bfaf-da25-4d88-a142-a15ee0000cf1","Type":"ContainerStarted","Data":"a89bc22d92ce4134a3496d22ac757da37711da519d3997532abf1387cf810230"} Jul 14 18:27:45.869342 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:45.869312 2569 generic.go:356] "Generic (PLEG): container finished" podID="0ed3bfaf-da25-4d88-a142-a15ee0000cf1" containerID="a89bc22d92ce4134a3496d22ac757da37711da519d3997532abf1387cf810230" exitCode=0 Jul 14 18:27:45.869747 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:45.869374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" event={"ID":"0ed3bfaf-da25-4d88-a142-a15ee0000cf1","Type":"ContainerDied","Data":"a89bc22d92ce4134a3496d22ac757da37711da519d3997532abf1387cf810230"} Jul 14 18:27:47.879465 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:47.879372 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" event={"ID":"0ed3bfaf-da25-4d88-a142-a15ee0000cf1","Type":"ContainerStarted","Data":"0e528cdb3dc7bfa2b30b4237035d2ac555a684b4d331123d56e75024c7092406"} Jul 14 18:27:47.879831 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:47.879498 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:47.952189 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:47.952136 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" podStartSLOduration=2.017836386 podStartE2EDuration="7.952121515s" totalImagesPullingTime="4.873963717s" totalInitContainerRuntime="1.060321412s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:40 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:27:47.951274341 +0000 UTC m=+362.008942400" watchObservedRunningTime="2026-07-14 18:27:47.952121515 +0000 UTC m=+362.009789621" Jul 14 18:27:58.885502 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:58.885472 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5d948cf9bc-c9r9f" Jul 14 18:27:59.143939 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.143858 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22"] Jul 14 18:27:59.153669 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.153642 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.158333 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.158307 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-99wmv\"" Jul 14 18:27:59.158670 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.158651 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 14 18:27:59.159446 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.159425 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 14 18:27:59.159543 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.159476 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 14 18:27:59.188432 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.188393 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22"] Jul 14 18:27:59.198150 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.198119 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z882\" (UniqueName: \"kubernetes.io/projected/351c593d-6f03-4821-9527-b21eb1aefec1-kube-api-access-4z882\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.198320 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.198181 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/351c593d-6f03-4821-9527-b21eb1aefec1-cert\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.198320 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.198208 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/351c593d-6f03-4821-9527-b21eb1aefec1-manager-config\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.198320 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.198228 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/351c593d-6f03-4821-9527-b21eb1aefec1-metrics-cert\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.299053 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.299020 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/351c593d-6f03-4821-9527-b21eb1aefec1-metrics-cert\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.299285 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.299080 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z882\" (UniqueName: \"kubernetes.io/projected/351c593d-6f03-4821-9527-b21eb1aefec1-kube-api-access-4z882\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.299285 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.299143 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/351c593d-6f03-4821-9527-b21eb1aefec1-cert\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.299285 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.299173 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/351c593d-6f03-4821-9527-b21eb1aefec1-manager-config\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.299868 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.299847 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/351c593d-6f03-4821-9527-b21eb1aefec1-manager-config\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.301647 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.301629 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/351c593d-6f03-4821-9527-b21eb1aefec1-cert\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.301743 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.301692 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/351c593d-6f03-4821-9527-b21eb1aefec1-metrics-cert\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.310666 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.310633 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z882\" (UniqueName: \"kubernetes.io/projected/351c593d-6f03-4821-9527-b21eb1aefec1-kube-api-access-4z882\") pod \"lws-controller-manager-8c5cc675c-m9p22\" (UID: \"351c593d-6f03-4821-9527-b21eb1aefec1\") " pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.463320 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.463196 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:27:59.620174 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.620140 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22"] Jul 14 18:27:59.622969 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:27:59.622945 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod351c593d_6f03_4821_9527_b21eb1aefec1.slice/crio-1941a3a0084c5977ad4f977c6542448cf70e448c0d96987bd86316153f106e0f WatchSource:0}: Error finding container 1941a3a0084c5977ad4f977c6542448cf70e448c0d96987bd86316153f106e0f: Status 404 returned error can't find the container with id 1941a3a0084c5977ad4f977c6542448cf70e448c0d96987bd86316153f106e0f Jul 14 18:27:59.929568 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:27:59.929535 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" event={"ID":"351c593d-6f03-4821-9527-b21eb1aefec1","Type":"ContainerStarted","Data":"1941a3a0084c5977ad4f977c6542448cf70e448c0d96987bd86316153f106e0f"} Jul 14 18:28:01.940460 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:01.940418 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" event={"ID":"351c593d-6f03-4821-9527-b21eb1aefec1","Type":"ContainerStarted","Data":"ea713707ddac85ad17a9694858bfaf94c42ce394edc3e5f09c62980d7dce5c08"} Jul 14 18:28:01.940851 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:01.940518 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:28:07.732981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.732930 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" podStartSLOduration=6.864720308 podStartE2EDuration="8.732911014s" totalImagesPullingTime="1.868190706s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:02.05186366 +0000 UTC m=+376.109532022" watchObservedRunningTime="2026-07-14 18:28:07.732911014 +0000 UTC m=+381.790579063" Jul 14 18:28:07.733385 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.733146 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7"] Jul 14 18:28:07.735814 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.735799 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:07.743191 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.743161 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-mw7nl\"" Jul 14 18:28:07.744815 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.744795 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 14 18:28:07.744932 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.744820 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 14 18:28:07.744932 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.744822 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 14 18:28:07.745292 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.745273 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 14 18:28:07.770562 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.770530 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7"] Jul 14 18:28:07.880600 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.880569 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ll4b9\" (UniqueName: \"kubernetes.io/projected/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-kube-api-access-ll4b9\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:07.880774 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.880611 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-tmp\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:07.880774 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.880714 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-tls-certs\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:07.982073 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.982040 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-tls-certs\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:07.982277 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.982086 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ll4b9\" (UniqueName: \"kubernetes.io/projected/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-kube-api-access-ll4b9\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:07.982277 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.982113 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-tmp\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:07.984391 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.984329 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-tmp\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:07.984561 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:07.984541 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-tls-certs\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:08.010494 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:08.010463 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ll4b9\" (UniqueName: \"kubernetes.io/projected/ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2-kube-api-access-ll4b9\") pod \"kube-auth-proxy-5d44f677bf-jj4k7\" (UID: \"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2\") " pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:08.047451 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:08.047418 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" Jul 14 18:28:08.195539 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:08.195509 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7"] Jul 14 18:28:08.197871 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:28:08.197848 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded27cfdc_d328_4b8e_a6e9_3b59b31c3ce2.slice/crio-1bc1b2d45a00e72aef16f8f8bef5ae40821506447778771005374ad4672c4759 WatchSource:0}: Error finding container 1bc1b2d45a00e72aef16f8f8bef5ae40821506447778771005374ad4672c4759: Status 404 returned error can't find the container with id 1bc1b2d45a00e72aef16f8f8bef5ae40821506447778771005374ad4672c4759 Jul 14 18:28:08.970233 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:08.970195 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" event={"ID":"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2","Type":"ContainerStarted","Data":"1bc1b2d45a00e72aef16f8f8bef5ae40821506447778771005374ad4672c4759"} Jul 14 18:28:12.946892 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:12.946862 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-8c5cc675c-m9p22" Jul 14 18:28:12.990731 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:28:12.990699 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" event={"ID":"ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2","Type":"ContainerStarted","Data":"a3d23ea9c1fd52247c10a8fbc2ca4adf1064a5a9065ebf8309e204b9a565c830"} Jul 14 18:29:20.559760 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.559659 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5d44f677bf-jj4k7" podStartSLOduration=69.417611509 podStartE2EDuration="1m13.559643003s" totalImagesPullingTime="4.142031494s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:28:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:28:13.203759329 +0000 UTC m=+387.261427390" watchObservedRunningTime="2026-07-14 18:29:20.559643003 +0000 UTC m=+454.617311054" Jul 14 18:29:20.560278 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.560247 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp"] Jul 14 18:29:20.563589 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.563572 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.571317 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.571298 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:29:20.572173 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.572159 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4g49d\"" Jul 14 18:29:20.572686 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.572673 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:29:20.578109 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.578090 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp"] Jul 14 18:29:20.662921 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.662879 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.663118 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.662931 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmz72\" (UniqueName: \"kubernetes.io/projected/3d2ab384-7cc6-4aa9-94b1-255c2c919320-kube-api-access-lmz72\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.663118 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.662994 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.764491 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.764453 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmz72\" (UniqueName: \"kubernetes.io/projected/3d2ab384-7cc6-4aa9-94b1-255c2c919320-kube-api-access-lmz72\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.764491 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.764497 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.764730 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.764568 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.764915 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.764894 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.764981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.764962 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.776003 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.775980 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmz72\" (UniqueName: \"kubernetes.io/projected/3d2ab384-7cc6-4aa9-94b1-255c2c919320-kube-api-access-lmz72\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:20.872220 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:20.872143 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:21.016242 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.016210 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp"] Jul 14 18:29:21.019312 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:29:21.019284 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d2ab384_7cc6_4aa9_94b1_255c2c919320.slice/crio-d42d141ceeb1f6ce80918e85716928fff399db21df56305c9225ea39dcafbdb3 WatchSource:0}: Error finding container d42d141ceeb1f6ce80918e85716928fff399db21df56305c9225ea39dcafbdb3: Status 404 returned error can't find the container with id d42d141ceeb1f6ce80918e85716928fff399db21df56305c9225ea39dcafbdb3 Jul 14 18:29:21.279317 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.279285 2569 generic.go:356] "Generic (PLEG): container finished" podID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerID="7acd75e26e8633dfbcd086226484c488cc201a22ffe5c73480fb0d8bf89da00f" exitCode=0 Jul 14 18:29:21.279496 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.279374 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" event={"ID":"3d2ab384-7cc6-4aa9-94b1-255c2c919320","Type":"ContainerDied","Data":"7acd75e26e8633dfbcd086226484c488cc201a22ffe5c73480fb0d8bf89da00f"} Jul 14 18:29:21.279496 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.279428 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" event={"ID":"3d2ab384-7cc6-4aa9-94b1-255c2c919320","Type":"ContainerStarted","Data":"d42d141ceeb1f6ce80918e85716928fff399db21df56305c9225ea39dcafbdb3"} Jul 14 18:29:21.335679 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.335646 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx"] Jul 14 18:29:21.338963 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.338941 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.356049 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.356013 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx"] Jul 14 18:29:21.469957 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.469876 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.469957 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.469937 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.470162 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.469989 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpln2\" (UniqueName: \"kubernetes.io/projected/61436e65-1689-4d0d-897c-e0285149edc6-kube-api-access-hpln2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.570995 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.570960 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.571477 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.571011 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpln2\" (UniqueName: \"kubernetes.io/projected/61436e65-1689-4d0d-897c-e0285149edc6-kube-api-access-hpln2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.571477 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.571061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.571477 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.571465 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.571617 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.571491 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.585147 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.585125 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpln2\" (UniqueName: \"kubernetes.io/projected/61436e65-1689-4d0d-897c-e0285149edc6-kube-api-access-hpln2\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.647418 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.647385 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:21.781632 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.781605 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx"] Jul 14 18:29:21.783089 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:29:21.783067 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61436e65_1689_4d0d_897c_e0285149edc6.slice/crio-651484322b9bf809cf1800f80b7ae8a9847b0c73a705c15a303fd1c3907907da WatchSource:0}: Error finding container 651484322b9bf809cf1800f80b7ae8a9847b0c73a705c15a303fd1c3907907da: Status 404 returned error can't find the container with id 651484322b9bf809cf1800f80b7ae8a9847b0c73a705c15a303fd1c3907907da Jul 14 18:29:21.926677 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.926642 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc"] Jul 14 18:29:21.929926 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.929909 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:21.953291 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:21.953247 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc"] Jul 14 18:29:22.075688 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.075639 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdrj4\" (UniqueName: \"kubernetes.io/projected/62a38bbb-557d-4a44-9442-0db22ca8bcd6-kube-api-access-kdrj4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.075688 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.075690 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.075918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.075720 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.176997 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.176964 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.177162 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.177061 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdrj4\" (UniqueName: \"kubernetes.io/projected/62a38bbb-557d-4a44-9442-0db22ca8bcd6-kube-api-access-kdrj4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.177162 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.177094 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.177494 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.177469 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.177494 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.177482 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.187073 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.187047 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdrj4\" (UniqueName: \"kubernetes.io/projected/62a38bbb-557d-4a44-9442-0db22ca8bcd6-kube-api-access-kdrj4\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.259209 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.259179 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:22.284680 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.284623 2569 generic.go:356] "Generic (PLEG): container finished" podID="61436e65-1689-4d0d-897c-e0285149edc6" containerID="28ab9146b8e0d61684033fdbb20f5e1bcaa7fa430c77337475a3b96ceb49ff58" exitCode=0 Jul 14 18:29:22.284680 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.284671 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" event={"ID":"61436e65-1689-4d0d-897c-e0285149edc6","Type":"ContainerDied","Data":"28ab9146b8e0d61684033fdbb20f5e1bcaa7fa430c77337475a3b96ceb49ff58"} Jul 14 18:29:22.284897 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.284715 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" event={"ID":"61436e65-1689-4d0d-897c-e0285149edc6","Type":"ContainerStarted","Data":"651484322b9bf809cf1800f80b7ae8a9847b0c73a705c15a303fd1c3907907da"} Jul 14 18:29:22.348920 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.348885 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb"] Jul 14 18:29:22.352541 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.352516 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.369511 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.369482 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb"] Jul 14 18:29:22.413461 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.413325 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc"] Jul 14 18:29:22.446481 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:29:22.446451 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a38bbb_557d_4a44_9442_0db22ca8bcd6.slice/crio-d92bdc699e82bae5ecc5e5ed95d33d083da12de1eb14e5affade3a1ed84784e4 WatchSource:0}: Error finding container d92bdc699e82bae5ecc5e5ed95d33d083da12de1eb14e5affade3a1ed84784e4: Status 404 returned error can't find the container with id d92bdc699e82bae5ecc5e5ed95d33d083da12de1eb14e5affade3a1ed84784e4 Jul 14 18:29:22.479577 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.479550 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srhw4\" (UniqueName: \"kubernetes.io/projected/cc7a3d39-6b7a-4005-a8f3-20523507a713-kube-api-access-srhw4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.479742 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.479598 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.479742 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.479622 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.580588 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.580562 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.580903 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.580596 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.580903 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.580651 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srhw4\" (UniqueName: \"kubernetes.io/projected/cc7a3d39-6b7a-4005-a8f3-20523507a713-kube-api-access-srhw4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.580903 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.580908 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.581028 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.580973 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.593416 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.593370 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srhw4\" (UniqueName: \"kubernetes.io/projected/cc7a3d39-6b7a-4005-a8f3-20523507a713-kube-api-access-srhw4\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.663373 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.663339 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:22.805994 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:22.805971 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb"] Jul 14 18:29:22.807777 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:29:22.807751 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc7a3d39_6b7a_4005_a8f3_20523507a713.slice/crio-47157b30a896b79058f523f0debaa45d9b3089cfa0f796aee9903baa60c40636 WatchSource:0}: Error finding container 47157b30a896b79058f523f0debaa45d9b3089cfa0f796aee9903baa60c40636: Status 404 returned error can't find the container with id 47157b30a896b79058f523f0debaa45d9b3089cfa0f796aee9903baa60c40636 Jul 14 18:29:23.291123 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:23.291087 2569 generic.go:356] "Generic (PLEG): container finished" podID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerID="b2d561a2aa14607371423394b384f7650f7b3e8f30fe2483a572a38657bea309" exitCode=0 Jul 14 18:29:23.291345 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:23.291187 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" event={"ID":"cc7a3d39-6b7a-4005-a8f3-20523507a713","Type":"ContainerDied","Data":"b2d561a2aa14607371423394b384f7650f7b3e8f30fe2483a572a38657bea309"} Jul 14 18:29:23.291345 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:23.291297 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" event={"ID":"cc7a3d39-6b7a-4005-a8f3-20523507a713","Type":"ContainerStarted","Data":"47157b30a896b79058f523f0debaa45d9b3089cfa0f796aee9903baa60c40636"} Jul 14 18:29:23.293117 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:23.293096 2569 generic.go:356] "Generic (PLEG): container finished" podID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerID="703e1b31d26a1bbfb727c2717bfeed41a1b35498c75f57fec091690eff3436ed" exitCode=0 Jul 14 18:29:23.293249 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:23.293175 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" event={"ID":"3d2ab384-7cc6-4aa9-94b1-255c2c919320","Type":"ContainerDied","Data":"703e1b31d26a1bbfb727c2717bfeed41a1b35498c75f57fec091690eff3436ed"} Jul 14 18:29:23.294730 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:23.294710 2569 generic.go:356] "Generic (PLEG): container finished" podID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerID="de3c8680e87836ce15027be101c836651846ee01d3c73958224f35e4844c32ae" exitCode=0 Jul 14 18:29:23.294799 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:23.294751 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" event={"ID":"62a38bbb-557d-4a44-9442-0db22ca8bcd6","Type":"ContainerDied","Data":"de3c8680e87836ce15027be101c836651846ee01d3c73958224f35e4844c32ae"} Jul 14 18:29:23.294799 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:23.294791 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" event={"ID":"62a38bbb-557d-4a44-9442-0db22ca8bcd6","Type":"ContainerStarted","Data":"d92bdc699e82bae5ecc5e5ed95d33d083da12de1eb14e5affade3a1ed84784e4"} Jul 14 18:29:24.304009 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:24.303980 2569 generic.go:356] "Generic (PLEG): container finished" podID="61436e65-1689-4d0d-897c-e0285149edc6" containerID="99d935bfc23a51b16daf52c93c267e4bfdbe5c1af61a60e4d1ba38e7f66a24ae" exitCode=0 Jul 14 18:29:24.304410 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:24.304064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" event={"ID":"61436e65-1689-4d0d-897c-e0285149edc6","Type":"ContainerDied","Data":"99d935bfc23a51b16daf52c93c267e4bfdbe5c1af61a60e4d1ba38e7f66a24ae"} Jul 14 18:29:24.306340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:24.306316 2569 generic.go:356] "Generic (PLEG): container finished" podID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerID="cf832e7b7800f791b8c7dc11bdf3dbbc9d45224f6696b83cfab5c8209bea924f" exitCode=0 Jul 14 18:29:24.306458 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:24.306351 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" event={"ID":"3d2ab384-7cc6-4aa9-94b1-255c2c919320","Type":"ContainerDied","Data":"cf832e7b7800f791b8c7dc11bdf3dbbc9d45224f6696b83cfab5c8209bea924f"} Jul 14 18:29:25.312954 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.312919 2569 generic.go:356] "Generic (PLEG): container finished" podID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerID="4e238c6bfade294496dbb5570ad9205a722a85b48825af114ecd3f9d1d62b8b7" exitCode=0 Jul 14 18:29:25.313398 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.312961 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" event={"ID":"62a38bbb-557d-4a44-9442-0db22ca8bcd6","Type":"ContainerDied","Data":"4e238c6bfade294496dbb5570ad9205a722a85b48825af114ecd3f9d1d62b8b7"} Jul 14 18:29:25.315145 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.315125 2569 generic.go:356] "Generic (PLEG): container finished" podID="61436e65-1689-4d0d-897c-e0285149edc6" containerID="50df48246dd74694271854c38a94055478781a601329e27d327c8571062d2633" exitCode=0 Jul 14 18:29:25.315232 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.315207 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" event={"ID":"61436e65-1689-4d0d-897c-e0285149edc6","Type":"ContainerDied","Data":"50df48246dd74694271854c38a94055478781a601329e27d327c8571062d2633"} Jul 14 18:29:25.316834 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.316817 2569 generic.go:356] "Generic (PLEG): container finished" podID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerID="a1925be6ec6d8b4b8c90b32237a01849c610c7a486c8d07e9188ab4fc87bf6ef" exitCode=0 Jul 14 18:29:25.316956 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.316935 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" event={"ID":"cc7a3d39-6b7a-4005-a8f3-20523507a713","Type":"ContainerDied","Data":"a1925be6ec6d8b4b8c90b32237a01849c610c7a486c8d07e9188ab4fc87bf6ef"} Jul 14 18:29:25.460407 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.460381 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:25.606363 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.606280 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-util\") pod \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " Jul 14 18:29:25.606506 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.606380 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lmz72\" (UniqueName: \"kubernetes.io/projected/3d2ab384-7cc6-4aa9-94b1-255c2c919320-kube-api-access-lmz72\") pod \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " Jul 14 18:29:25.606506 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.606423 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-bundle\") pod \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\" (UID: \"3d2ab384-7cc6-4aa9-94b1-255c2c919320\") " Jul 14 18:29:25.606854 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.606831 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-bundle" (OuterVolumeSpecName: "bundle") pod "3d2ab384-7cc6-4aa9-94b1-255c2c919320" (UID: "3d2ab384-7cc6-4aa9-94b1-255c2c919320"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:29:25.608638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.608610 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d2ab384-7cc6-4aa9-94b1-255c2c919320-kube-api-access-lmz72" (OuterVolumeSpecName: "kube-api-access-lmz72") pod "3d2ab384-7cc6-4aa9-94b1-255c2c919320" (UID: "3d2ab384-7cc6-4aa9-94b1-255c2c919320"). InnerVolumeSpecName "kube-api-access-lmz72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:29:25.612023 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.612004 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-util" (OuterVolumeSpecName: "util") pod "3d2ab384-7cc6-4aa9-94b1-255c2c919320" (UID: "3d2ab384-7cc6-4aa9-94b1-255c2c919320"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:29:25.707147 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.707115 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmz72\" (UniqueName: \"kubernetes.io/projected/3d2ab384-7cc6-4aa9-94b1-255c2c919320-kube-api-access-lmz72\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:25.707147 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.707144 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:25.707147 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:25.707156 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3d2ab384-7cc6-4aa9-94b1-255c2c919320-util\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:26.322892 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.322863 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" Jul 14 18:29:26.322892 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.322870 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp" event={"ID":"3d2ab384-7cc6-4aa9-94b1-255c2c919320","Type":"ContainerDied","Data":"d42d141ceeb1f6ce80918e85716928fff399db21df56305c9225ea39dcafbdb3"} Jul 14 18:29:26.323424 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.322947 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d42d141ceeb1f6ce80918e85716928fff399db21df56305c9225ea39dcafbdb3" Jul 14 18:29:26.325066 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.325038 2569 generic.go:356] "Generic (PLEG): container finished" podID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerID="ca698bab44f55922493c541e12b038b12bbee59cd48643e0a7a4d1aea1397aae" exitCode=0 Jul 14 18:29:26.325205 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.325066 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" event={"ID":"62a38bbb-557d-4a44-9442-0db22ca8bcd6","Type":"ContainerDied","Data":"ca698bab44f55922493c541e12b038b12bbee59cd48643e0a7a4d1aea1397aae"} Jul 14 18:29:26.327307 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.327285 2569 generic.go:356] "Generic (PLEG): container finished" podID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerID="e2cd06214e2c62923adce3bbf0794ceea95d610c5a0780076645b1da72d0a289" exitCode=0 Jul 14 18:29:26.327417 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.327355 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" event={"ID":"cc7a3d39-6b7a-4005-a8f3-20523507a713","Type":"ContainerDied","Data":"e2cd06214e2c62923adce3bbf0794ceea95d610c5a0780076645b1da72d0a289"} Jul 14 18:29:26.453925 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.453903 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:26.617146 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.617053 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-bundle\") pod \"61436e65-1689-4d0d-897c-e0285149edc6\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " Jul 14 18:29:26.617146 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.617131 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-util\") pod \"61436e65-1689-4d0d-897c-e0285149edc6\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " Jul 14 18:29:26.617382 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.617194 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpln2\" (UniqueName: \"kubernetes.io/projected/61436e65-1689-4d0d-897c-e0285149edc6-kube-api-access-hpln2\") pod \"61436e65-1689-4d0d-897c-e0285149edc6\" (UID: \"61436e65-1689-4d0d-897c-e0285149edc6\") " Jul 14 18:29:26.617931 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.617901 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-bundle" (OuterVolumeSpecName: "bundle") pod "61436e65-1689-4d0d-897c-e0285149edc6" (UID: "61436e65-1689-4d0d-897c-e0285149edc6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:29:26.619771 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.619743 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61436e65-1689-4d0d-897c-e0285149edc6-kube-api-access-hpln2" (OuterVolumeSpecName: "kube-api-access-hpln2") pod "61436e65-1689-4d0d-897c-e0285149edc6" (UID: "61436e65-1689-4d0d-897c-e0285149edc6"). InnerVolumeSpecName "kube-api-access-hpln2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:29:26.623233 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.623200 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-util" (OuterVolumeSpecName: "util") pod "61436e65-1689-4d0d-897c-e0285149edc6" (UID: "61436e65-1689-4d0d-897c-e0285149edc6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:29:26.718900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.718863 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:26.718900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.718893 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/61436e65-1689-4d0d-897c-e0285149edc6-util\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:26.718900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:26.718903 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpln2\" (UniqueName: \"kubernetes.io/projected/61436e65-1689-4d0d-897c-e0285149edc6-kube-api-access-hpln2\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:27.333110 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.333083 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" Jul 14 18:29:27.333497 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.333110 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx" event={"ID":"61436e65-1689-4d0d-897c-e0285149edc6","Type":"ContainerDied","Data":"651484322b9bf809cf1800f80b7ae8a9847b0c73a705c15a303fd1c3907907da"} Jul 14 18:29:27.333497 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.333156 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="651484322b9bf809cf1800f80b7ae8a9847b0c73a705c15a303fd1c3907907da" Jul 14 18:29:27.458971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.458948 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:27.490465 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.490440 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:29:27.627625 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.627536 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-util\") pod \"cc7a3d39-6b7a-4005-a8f3-20523507a713\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " Jul 14 18:29:27.627625 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.627605 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-bundle\") pod \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " Jul 14 18:29:27.627843 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.627638 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-util\") pod \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " Jul 14 18:29:27.627843 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.627666 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-srhw4\" (UniqueName: \"kubernetes.io/projected/cc7a3d39-6b7a-4005-a8f3-20523507a713-kube-api-access-srhw4\") pod \"cc7a3d39-6b7a-4005-a8f3-20523507a713\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " Jul 14 18:29:27.627843 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.627702 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kdrj4\" (UniqueName: \"kubernetes.io/projected/62a38bbb-557d-4a44-9442-0db22ca8bcd6-kube-api-access-kdrj4\") pod \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\" (UID: \"62a38bbb-557d-4a44-9442-0db22ca8bcd6\") " Jul 14 18:29:27.627843 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.627733 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-bundle\") pod \"cc7a3d39-6b7a-4005-a8f3-20523507a713\" (UID: \"cc7a3d39-6b7a-4005-a8f3-20523507a713\") " Jul 14 18:29:27.628714 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.628476 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-bundle" (OuterVolumeSpecName: "bundle") pod "cc7a3d39-6b7a-4005-a8f3-20523507a713" (UID: "cc7a3d39-6b7a-4005-a8f3-20523507a713"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:29:27.628885 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.628841 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-bundle" (OuterVolumeSpecName: "bundle") pod "62a38bbb-557d-4a44-9442-0db22ca8bcd6" (UID: "62a38bbb-557d-4a44-9442-0db22ca8bcd6"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:29:27.630022 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.629986 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a38bbb-557d-4a44-9442-0db22ca8bcd6-kube-api-access-kdrj4" (OuterVolumeSpecName: "kube-api-access-kdrj4") pod "62a38bbb-557d-4a44-9442-0db22ca8bcd6" (UID: "62a38bbb-557d-4a44-9442-0db22ca8bcd6"). InnerVolumeSpecName "kube-api-access-kdrj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:29:27.630153 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.630068 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc7a3d39-6b7a-4005-a8f3-20523507a713-kube-api-access-srhw4" (OuterVolumeSpecName: "kube-api-access-srhw4") pod "cc7a3d39-6b7a-4005-a8f3-20523507a713" (UID: "cc7a3d39-6b7a-4005-a8f3-20523507a713"). InnerVolumeSpecName "kube-api-access-srhw4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:29:27.634036 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.634008 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-util" (OuterVolumeSpecName: "util") pod "cc7a3d39-6b7a-4005-a8f3-20523507a713" (UID: "cc7a3d39-6b7a-4005-a8f3-20523507a713"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:29:27.634332 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.634313 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-util" (OuterVolumeSpecName: "util") pod "62a38bbb-557d-4a44-9442-0db22ca8bcd6" (UID: "62a38bbb-557d-4a44-9442-0db22ca8bcd6"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:29:27.728631 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.728595 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-util\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:27.728631 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.728628 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:27.728631 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.728636 2569 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/62a38bbb-557d-4a44-9442-0db22ca8bcd6-util\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:27.728858 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.728645 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srhw4\" (UniqueName: \"kubernetes.io/projected/cc7a3d39-6b7a-4005-a8f3-20523507a713-kube-api-access-srhw4\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:27.728858 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.728656 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdrj4\" (UniqueName: \"kubernetes.io/projected/62a38bbb-557d-4a44-9442-0db22ca8bcd6-kube-api-access-kdrj4\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:27.728858 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:27.728664 2569 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/cc7a3d39-6b7a-4005-a8f3-20523507a713-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:29:28.340067 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:28.340037 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" Jul 14 18:29:28.340067 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:28.340053 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb" event={"ID":"cc7a3d39-6b7a-4005-a8f3-20523507a713","Type":"ContainerDied","Data":"47157b30a896b79058f523f0debaa45d9b3089cfa0f796aee9903baa60c40636"} Jul 14 18:29:28.340591 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:28.340124 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="47157b30a896b79058f523f0debaa45d9b3089cfa0f796aee9903baa60c40636" Jul 14 18:29:28.342003 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:28.341978 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" event={"ID":"62a38bbb-557d-4a44-9442-0db22ca8bcd6","Type":"ContainerDied","Data":"d92bdc699e82bae5ecc5e5ed95d33d083da12de1eb14e5affade3a1ed84784e4"} Jul 14 18:29:28.342163 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:28.342013 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d92bdc699e82bae5ecc5e5ed95d33d083da12de1eb14e5affade3a1ed84784e4" Jul 14 18:29:28.342163 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:29:28.342023 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc" Jul 14 18:30:02.191379 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191346 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7"] Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191606 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerName="util" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191617 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerName="util" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191628 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191633 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191642 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61436e65-1689-4d0d-897c-e0285149edc6" containerName="pull" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191647 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="61436e65-1689-4d0d-897c-e0285149edc6" containerName="pull" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191656 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61436e65-1689-4d0d-897c-e0285149edc6" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191661 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="61436e65-1689-4d0d-897c-e0285149edc6" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191669 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerName="pull" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191674 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerName="pull" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191680 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerName="pull" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191685 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerName="pull" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191692 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerName="pull" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191697 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerName="pull" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191704 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerName="util" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191709 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerName="util" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191714 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191719 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191727 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerName="util" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191731 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerName="util" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191738 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61436e65-1689-4d0d-897c-e0285149edc6" containerName="util" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191742 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="61436e65-1689-4d0d-897c-e0285149edc6" containerName="util" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191748 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191752 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191793 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a38bbb-557d-4a44-9442-0db22ca8bcd6" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191799 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc7a3d39-6b7a-4005-a8f3-20523507a713" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191805 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d2ab384-7cc6-4aa9-94b1-255c2c919320" containerName="extract" Jul 14 18:30:02.191848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.191810 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="61436e65-1689-4d0d-897c-e0285149edc6" containerName="extract" Jul 14 18:30:02.196617 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.196598 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.199725 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.199705 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 14 18:30:02.200221 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.200205 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 14 18:30:02.201119 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.201104 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-4g49d\"" Jul 14 18:30:02.201119 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.201112 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 14 18:30:02.201275 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.201161 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 14 18:30:02.213019 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.212999 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7"] Jul 14 18:30:02.309078 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.309038 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klzsp\" (UniqueName: \"kubernetes.io/projected/9a665233-2e99-427f-9197-bc5408c47dc2-kube-api-access-klzsp\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.309288 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.309107 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a665233-2e99-427f-9197-bc5408c47dc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.309288 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.309185 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a665233-2e99-427f-9197-bc5408c47dc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.410417 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.410380 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klzsp\" (UniqueName: \"kubernetes.io/projected/9a665233-2e99-427f-9197-bc5408c47dc2-kube-api-access-klzsp\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.410572 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.410542 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a665233-2e99-427f-9197-bc5408c47dc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.410637 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.410579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a665233-2e99-427f-9197-bc5408c47dc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.411169 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.411151 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9a665233-2e99-427f-9197-bc5408c47dc2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.412935 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.412907 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/9a665233-2e99-427f-9197-bc5408c47dc2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.423657 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.423627 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-klzsp\" (UniqueName: \"kubernetes.io/projected/9a665233-2e99-427f-9197-bc5408c47dc2-kube-api-access-klzsp\") pod \"kuadrant-console-plugin-66b5774985-q5hv7\" (UID: \"9a665233-2e99-427f-9197-bc5408c47dc2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.505222 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.505194 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" Jul 14 18:30:02.642045 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:02.642022 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7"] Jul 14 18:30:02.644599 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:30:02.644573 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a665233_2e99_427f_9197_bc5408c47dc2.slice/crio-3e8748849b41570617b065719233a609a07ad330dda78ffe33d63601442daa76 WatchSource:0}: Error finding container 3e8748849b41570617b065719233a609a07ad330dda78ffe33d63601442daa76: Status 404 returned error can't find the container with id 3e8748849b41570617b065719233a609a07ad330dda78ffe33d63601442daa76 Jul 14 18:30:03.502896 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:03.502860 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" event={"ID":"9a665233-2e99-427f-9197-bc5408c47dc2","Type":"ContainerStarted","Data":"3e8748849b41570617b065719233a609a07ad330dda78ffe33d63601442daa76"} Jul 14 18:30:28.635837 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:28.635797 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" event={"ID":"9a665233-2e99-427f-9197-bc5408c47dc2","Type":"ContainerStarted","Data":"7d5ed61333e365c68d3fff9387e2e1d6f01dca00c945127d2fe642aa8baf1e03"} Jul 14 18:30:28.669163 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:30:28.669114 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-q5hv7" podStartSLOduration=1.709216279 podStartE2EDuration="26.669098837s" totalImagesPullingTime="24.959882558s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:30:28.667187358 +0000 UTC m=+522.724855432" watchObservedRunningTime="2026-07-14 18:30:28.669098837 +0000 UTC m=+522.726766885" Jul 14 18:31:00.749223 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.749133 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:31:00.814461 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.814429 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:31:00.814461 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.814468 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:31:00.814655 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.814540 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:00.822008 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.821990 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 14 18:31:00.898735 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.898696 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09ffa02e-e0d5-4ecb-98cb-5e4907efd905-config-file\") pod \"limitador-limitador-54d6d988d5-8cxq5\" (UID: \"09ffa02e-e0d5-4ecb-98cb-5e4907efd905\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:00.898735 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.898744 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4j7l\" (UniqueName: \"kubernetes.io/projected/09ffa02e-e0d5-4ecb-98cb-5e4907efd905-kube-api-access-t4j7l\") pod \"limitador-limitador-54d6d988d5-8cxq5\" (UID: \"09ffa02e-e0d5-4ecb-98cb-5e4907efd905\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:01.000031 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.999935 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09ffa02e-e0d5-4ecb-98cb-5e4907efd905-config-file\") pod \"limitador-limitador-54d6d988d5-8cxq5\" (UID: \"09ffa02e-e0d5-4ecb-98cb-5e4907efd905\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:01.000031 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:00.999976 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4j7l\" (UniqueName: \"kubernetes.io/projected/09ffa02e-e0d5-4ecb-98cb-5e4907efd905-kube-api-access-t4j7l\") pod \"limitador-limitador-54d6d988d5-8cxq5\" (UID: \"09ffa02e-e0d5-4ecb-98cb-5e4907efd905\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:01.000610 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.000588 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/09ffa02e-e0d5-4ecb-98cb-5e4907efd905-config-file\") pod \"limitador-limitador-54d6d988d5-8cxq5\" (UID: \"09ffa02e-e0d5-4ecb-98cb-5e4907efd905\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:01.014184 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.014158 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4j7l\" (UniqueName: \"kubernetes.io/projected/09ffa02e-e0d5-4ecb-98cb-5e4907efd905-kube-api-access-t4j7l\") pod \"limitador-limitador-54d6d988d5-8cxq5\" (UID: \"09ffa02e-e0d5-4ecb-98cb-5e4907efd905\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:01.123738 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.123703 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:01.171355 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.171229 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-jcsfb"] Jul 14 18:31:01.174297 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.174271 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jcsfb" Jul 14 18:31:01.181380 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.181355 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z7rpx\"" Jul 14 18:31:01.214067 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.214034 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jcsfb"] Jul 14 18:31:01.281541 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.281477 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:31:01.284009 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:31:01.283971 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ffa02e_e0d5_4ecb_98cb_5e4907efd905.slice/crio-16969a427480592d83c1ce7c60f1c4b9df576aa268e40d20bcf3a46aa9d74019 WatchSource:0}: Error finding container 16969a427480592d83c1ce7c60f1c4b9df576aa268e40d20bcf3a46aa9d74019: Status 404 returned error can't find the container with id 16969a427480592d83c1ce7c60f1c4b9df576aa268e40d20bcf3a46aa9d74019 Jul 14 18:31:01.303270 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.303234 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnl6q\" (UniqueName: \"kubernetes.io/projected/8a3cf96a-44aa-4821-a39a-309e173194b2-kube-api-access-wnl6q\") pod \"authorino-58cd955675-jcsfb\" (UID: \"8a3cf96a-44aa-4821-a39a-309e173194b2\") " pod="kuadrant-system/authorino-58cd955675-jcsfb" Jul 14 18:31:01.404431 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.404397 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnl6q\" (UniqueName: \"kubernetes.io/projected/8a3cf96a-44aa-4821-a39a-309e173194b2-kube-api-access-wnl6q\") pod \"authorino-58cd955675-jcsfb\" (UID: \"8a3cf96a-44aa-4821-a39a-309e173194b2\") " pod="kuadrant-system/authorino-58cd955675-jcsfb" Jul 14 18:31:01.415234 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.415206 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnl6q\" (UniqueName: \"kubernetes.io/projected/8a3cf96a-44aa-4821-a39a-309e173194b2-kube-api-access-wnl6q\") pod \"authorino-58cd955675-jcsfb\" (UID: \"8a3cf96a-44aa-4821-a39a-309e173194b2\") " pod="kuadrant-system/authorino-58cd955675-jcsfb" Jul 14 18:31:01.486916 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.486875 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jcsfb" Jul 14 18:31:01.639485 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.639455 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-jcsfb"] Jul 14 18:31:01.640413 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:31:01.640383 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8a3cf96a_44aa_4821_a39a_309e173194b2.slice/crio-05cae85fd2dc44503973f2bf823a100aca7fc0bf35ebf07a790d7e50f94cd8e8 WatchSource:0}: Error finding container 05cae85fd2dc44503973f2bf823a100aca7fc0bf35ebf07a790d7e50f94cd8e8: Status 404 returned error can't find the container with id 05cae85fd2dc44503973f2bf823a100aca7fc0bf35ebf07a790d7e50f94cd8e8 Jul 14 18:31:01.785063 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.784968 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jcsfb" event={"ID":"8a3cf96a-44aa-4821-a39a-309e173194b2","Type":"ContainerStarted","Data":"05cae85fd2dc44503973f2bf823a100aca7fc0bf35ebf07a790d7e50f94cd8e8"} Jul 14 18:31:01.786096 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:01.786074 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" event={"ID":"09ffa02e-e0d5-4ecb-98cb-5e4907efd905","Type":"ContainerStarted","Data":"16969a427480592d83c1ce7c60f1c4b9df576aa268e40d20bcf3a46aa9d74019"} Jul 14 18:31:06.213264 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:06.213216 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jcsfb"] Jul 14 18:31:06.824113 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:06.824064 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jcsfb" event={"ID":"8a3cf96a-44aa-4821-a39a-309e173194b2","Type":"ContainerStarted","Data":"79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd"} Jul 14 18:31:06.825693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:06.825616 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" event={"ID":"09ffa02e-e0d5-4ecb-98cb-5e4907efd905","Type":"ContainerStarted","Data":"ba2dcf9ed4e20c054bbce4d2b9438cc4f9453b7c977c5117d1fa2c2531a5fd63"} Jul 14 18:31:06.825819 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:06.825732 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:06.846553 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:06.846505 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-jcsfb" podStartSLOduration=1.4272909999999999 podStartE2EDuration="5.846491609s" totalImagesPullingTime="4.419200609s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:06.844232054 +0000 UTC m=+560.901900118" watchObservedRunningTime="2026-07-14 18:31:06.846491609 +0000 UTC m=+560.904159656" Jul 14 18:31:06.872429 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:06.872374 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" podStartSLOduration=2.09059467 podStartE2EDuration="6.872357958s" totalImagesPullingTime="4.781763288s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:06.872170041 +0000 UTC m=+560.929838101" watchObservedRunningTime="2026-07-14 18:31:06.872357958 +0000 UTC m=+560.930026039" Jul 14 18:31:07.830604 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:07.830563 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-jcsfb" podUID="8a3cf96a-44aa-4821-a39a-309e173194b2" containerName="authorino" containerID="cri-o://79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd" gracePeriod=30 Jul 14 18:31:08.085500 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.085437 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jcsfb" Jul 14 18:31:08.167067 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.167030 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnl6q\" (UniqueName: \"kubernetes.io/projected/8a3cf96a-44aa-4821-a39a-309e173194b2-kube-api-access-wnl6q\") pod \"8a3cf96a-44aa-4821-a39a-309e173194b2\" (UID: \"8a3cf96a-44aa-4821-a39a-309e173194b2\") " Jul 14 18:31:08.169191 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.169168 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3cf96a-44aa-4821-a39a-309e173194b2-kube-api-access-wnl6q" (OuterVolumeSpecName: "kube-api-access-wnl6q") pod "8a3cf96a-44aa-4821-a39a-309e173194b2" (UID: "8a3cf96a-44aa-4821-a39a-309e173194b2"). InnerVolumeSpecName "kube-api-access-wnl6q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:31:08.267856 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.267821 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnl6q\" (UniqueName: \"kubernetes.io/projected/8a3cf96a-44aa-4821-a39a-309e173194b2-kube-api-access-wnl6q\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:31:08.835902 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.835865 2569 generic.go:356] "Generic (PLEG): container finished" podID="8a3cf96a-44aa-4821-a39a-309e173194b2" containerID="79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd" exitCode=0 Jul 14 18:31:08.836400 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.835925 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-jcsfb" Jul 14 18:31:08.836400 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.835939 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jcsfb" event={"ID":"8a3cf96a-44aa-4821-a39a-309e173194b2","Type":"ContainerDied","Data":"79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd"} Jul 14 18:31:08.836400 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.835969 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-jcsfb" event={"ID":"8a3cf96a-44aa-4821-a39a-309e173194b2","Type":"ContainerDied","Data":"05cae85fd2dc44503973f2bf823a100aca7fc0bf35ebf07a790d7e50f94cd8e8"} Jul 14 18:31:08.836400 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.835984 2569 scope.go:117] "RemoveContainer" containerID="79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd" Jul 14 18:31:08.844313 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.844297 2569 scope.go:117] "RemoveContainer" containerID="79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd" Jul 14 18:31:08.844587 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:31:08.844569 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd\": container with ID starting with 79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd not found: ID does not exist" containerID="79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd" Jul 14 18:31:08.844653 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.844599 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd"} err="failed to get container status \"79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd\": rpc error: code = NotFound desc = could not find container \"79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd\": container with ID starting with 79d2eba457c5e32abc9e73e32f0ec048e70085804eb5b09ba1c927497cd128cd not found: ID does not exist" Jul 14 18:31:08.864858 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.864830 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-jcsfb"] Jul 14 18:31:08.877150 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:08.877122 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-jcsfb"] Jul 14 18:31:10.482069 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:10.482035 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a3cf96a-44aa-4821-a39a-309e173194b2" path="/var/lib/kubelet/pods/8a3cf96a-44aa-4821-a39a-309e173194b2/volumes" Jul 14 18:31:17.831637 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:17.831606 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-8cxq5" Jul 14 18:31:34.047860 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.047817 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlhf4"] Jul 14 18:31:34.048285 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.048164 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a3cf96a-44aa-4821-a39a-309e173194b2" containerName="authorino" Jul 14 18:31:34.048285 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.048179 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a3cf96a-44aa-4821-a39a-309e173194b2" containerName="authorino" Jul 14 18:31:34.048285 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.048240 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a3cf96a-44aa-4821-a39a-309e173194b2" containerName="authorino" Jul 14 18:31:34.051091 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.051073 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" Jul 14 18:31:34.056476 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.056450 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z7rpx\"" Jul 14 18:31:34.075227 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.075194 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlhf4"] Jul 14 18:31:34.077465 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.077440 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xw9h\" (UniqueName: \"kubernetes.io/projected/48052ae2-569f-4dd6-8f43-5aeafa311a82-kube-api-access-2xw9h\") pod \"authorino-5d4b67c446-mlhf4\" (UID: \"48052ae2-569f-4dd6-8f43-5aeafa311a82\") " pod="kuadrant-system/authorino-5d4b67c446-mlhf4" Jul 14 18:31:34.178208 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.178170 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xw9h\" (UniqueName: \"kubernetes.io/projected/48052ae2-569f-4dd6-8f43-5aeafa311a82-kube-api-access-2xw9h\") pod \"authorino-5d4b67c446-mlhf4\" (UID: \"48052ae2-569f-4dd6-8f43-5aeafa311a82\") " pod="kuadrant-system/authorino-5d4b67c446-mlhf4" Jul 14 18:31:34.193826 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.193795 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xw9h\" (UniqueName: \"kubernetes.io/projected/48052ae2-569f-4dd6-8f43-5aeafa311a82-kube-api-access-2xw9h\") pod \"authorino-5d4b67c446-mlhf4\" (UID: \"48052ae2-569f-4dd6-8f43-5aeafa311a82\") " pod="kuadrant-system/authorino-5d4b67c446-mlhf4" Jul 14 18:31:34.309519 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.309434 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlhf4"] Jul 14 18:31:34.309678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.309667 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" Jul 14 18:31:34.385607 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.385573 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6455c599f7-hq2cg"] Jul 14 18:31:34.387972 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.387951 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6455c599f7-hq2cg" Jul 14 18:31:34.403519 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.403488 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6455c599f7-hq2cg"] Jul 14 18:31:34.470957 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.470935 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlhf4"] Jul 14 18:31:34.473697 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:31:34.473667 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48052ae2_569f_4dd6_8f43_5aeafa311a82.slice/crio-14afc09f3957c46722255623772753e24600ba8de31bd6c8058c30c76b04733a WatchSource:0}: Error finding container 14afc09f3957c46722255623772753e24600ba8de31bd6c8058c30c76b04733a: Status 404 returned error can't find the container with id 14afc09f3957c46722255623772753e24600ba8de31bd6c8058c30c76b04733a Jul 14 18:31:34.479890 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.479844 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b45j5\" (UniqueName: \"kubernetes.io/projected/15de8ed9-a5cb-4d67-8e8e-2fb84accae85-kube-api-access-b45j5\") pod \"authorino-6455c599f7-hq2cg\" (UID: \"15de8ed9-a5cb-4d67-8e8e-2fb84accae85\") " pod="kuadrant-system/authorino-6455c599f7-hq2cg" Jul 14 18:31:34.580521 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.580426 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b45j5\" (UniqueName: \"kubernetes.io/projected/15de8ed9-a5cb-4d67-8e8e-2fb84accae85-kube-api-access-b45j5\") pod \"authorino-6455c599f7-hq2cg\" (UID: \"15de8ed9-a5cb-4d67-8e8e-2fb84accae85\") " pod="kuadrant-system/authorino-6455c599f7-hq2cg" Jul 14 18:31:34.593454 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.593430 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b45j5\" (UniqueName: \"kubernetes.io/projected/15de8ed9-a5cb-4d67-8e8e-2fb84accae85-kube-api-access-b45j5\") pod \"authorino-6455c599f7-hq2cg\" (UID: \"15de8ed9-a5cb-4d67-8e8e-2fb84accae85\") " pod="kuadrant-system/authorino-6455c599f7-hq2cg" Jul 14 18:31:34.671112 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.671073 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6455c599f7-hq2cg"] Jul 14 18:31:34.671332 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.671318 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6455c599f7-hq2cg" Jul 14 18:31:34.828819 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.828664 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6455c599f7-hq2cg"] Jul 14 18:31:34.830916 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:31:34.830851 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod15de8ed9_a5cb_4d67_8e8e_2fb84accae85.slice/crio-a6db9c389eddbcbfa58c2cf787eaa90422858a4a89effa67a94a3e76d783c270 WatchSource:0}: Error finding container a6db9c389eddbcbfa58c2cf787eaa90422858a4a89effa67a94a3e76d783c270: Status 404 returned error can't find the container with id a6db9c389eddbcbfa58c2cf787eaa90422858a4a89effa67a94a3e76d783c270 Jul 14 18:31:34.957995 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.957954 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" event={"ID":"48052ae2-569f-4dd6-8f43-5aeafa311a82","Type":"ContainerStarted","Data":"a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340"} Jul 14 18:31:34.957995 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.957970 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" podUID="48052ae2-569f-4dd6-8f43-5aeafa311a82" containerName="authorino" containerID="cri-o://a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340" gracePeriod=30 Jul 14 18:31:34.958303 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.958007 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" event={"ID":"48052ae2-569f-4dd6-8f43-5aeafa311a82","Type":"ContainerStarted","Data":"14afc09f3957c46722255623772753e24600ba8de31bd6c8058c30c76b04733a"} Jul 14 18:31:34.959244 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.959218 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6455c599f7-hq2cg" event={"ID":"15de8ed9-a5cb-4d67-8e8e-2fb84accae85","Type":"ContainerStarted","Data":"a6db9c389eddbcbfa58c2cf787eaa90422858a4a89effa67a94a3e76d783c270"} Jul 14 18:31:34.986713 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:34.986665 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" podStartSLOduration=0.570706613 podStartE2EDuration="986.648632ms" totalImagesPullingTime="415.942019ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:34.985625549 +0000 UTC m=+589.043293616" watchObservedRunningTime="2026-07-14 18:31:34.986648632 +0000 UTC m=+589.044316681" Jul 14 18:31:35.184656 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.184631 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" Jul 14 18:31:35.287129 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.287103 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xw9h\" (UniqueName: \"kubernetes.io/projected/48052ae2-569f-4dd6-8f43-5aeafa311a82-kube-api-access-2xw9h\") pod \"48052ae2-569f-4dd6-8f43-5aeafa311a82\" (UID: \"48052ae2-569f-4dd6-8f43-5aeafa311a82\") " Jul 14 18:31:35.289387 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.289353 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48052ae2-569f-4dd6-8f43-5aeafa311a82-kube-api-access-2xw9h" (OuterVolumeSpecName: "kube-api-access-2xw9h") pod "48052ae2-569f-4dd6-8f43-5aeafa311a82" (UID: "48052ae2-569f-4dd6-8f43-5aeafa311a82"). InnerVolumeSpecName "kube-api-access-2xw9h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:31:35.388227 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.388134 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xw9h\" (UniqueName: \"kubernetes.io/projected/48052ae2-569f-4dd6-8f43-5aeafa311a82-kube-api-access-2xw9h\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:31:35.970519 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.970485 2569 generic.go:356] "Generic (PLEG): container finished" podID="48052ae2-569f-4dd6-8f43-5aeafa311a82" containerID="a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340" exitCode=2 Jul 14 18:31:35.970716 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.970555 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" Jul 14 18:31:35.970716 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.970574 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" event={"ID":"48052ae2-569f-4dd6-8f43-5aeafa311a82","Type":"ContainerDied","Data":"a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340"} Jul 14 18:31:35.970716 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.970610 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-mlhf4" event={"ID":"48052ae2-569f-4dd6-8f43-5aeafa311a82","Type":"ContainerDied","Data":"14afc09f3957c46722255623772753e24600ba8de31bd6c8058c30c76b04733a"} Jul 14 18:31:35.970716 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.970627 2569 scope.go:117] "RemoveContainer" containerID="a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340" Jul 14 18:31:35.972189 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.972166 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6455c599f7-hq2cg" event={"ID":"15de8ed9-a5cb-4d67-8e8e-2fb84accae85","Type":"ContainerStarted","Data":"23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0"} Jul 14 18:31:35.972357 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.972337 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-6455c599f7-hq2cg" podUID="15de8ed9-a5cb-4d67-8e8e-2fb84accae85" containerName="authorino" containerID="cri-o://23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0" gracePeriod=30 Jul 14 18:31:35.980445 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.980422 2569 scope.go:117] "RemoveContainer" containerID="a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340" Jul 14 18:31:35.980761 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:31:35.980741 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340\": container with ID starting with a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340 not found: ID does not exist" containerID="a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340" Jul 14 18:31:35.980821 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:35.980772 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340"} err="failed to get container status \"a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340\": rpc error: code = NotFound desc = could not find container \"a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340\": container with ID starting with a3bf55b645511403ca548426144a04e52df3a9bba6ce17ebd5adeb3d779b8340 not found: ID does not exist" Jul 14 18:31:36.011584 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.011532 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6455c599f7-hq2cg" podStartSLOduration=1.6091770300000001 podStartE2EDuration="2.011514535s" totalImagesPullingTime="402.337505ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:31:36.01018272 +0000 UTC m=+590.067850786" watchObservedRunningTime="2026-07-14 18:31:36.011514535 +0000 UTC m=+590.069182583" Jul 14 18:31:36.049670 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.049639 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlhf4"] Jul 14 18:31:36.059669 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.059638 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-mlhf4"] Jul 14 18:31:36.216305 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.216276 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6455c599f7-hq2cg" Jul 14 18:31:36.296805 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.296767 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b45j5\" (UniqueName: \"kubernetes.io/projected/15de8ed9-a5cb-4d67-8e8e-2fb84accae85-kube-api-access-b45j5\") pod \"15de8ed9-a5cb-4d67-8e8e-2fb84accae85\" (UID: \"15de8ed9-a5cb-4d67-8e8e-2fb84accae85\") " Jul 14 18:31:36.298875 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.298845 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15de8ed9-a5cb-4d67-8e8e-2fb84accae85-kube-api-access-b45j5" (OuterVolumeSpecName: "kube-api-access-b45j5") pod "15de8ed9-a5cb-4d67-8e8e-2fb84accae85" (UID: "15de8ed9-a5cb-4d67-8e8e-2fb84accae85"). InnerVolumeSpecName "kube-api-access-b45j5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:31:36.398327 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.398283 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b45j5\" (UniqueName: \"kubernetes.io/projected/15de8ed9-a5cb-4d67-8e8e-2fb84accae85-kube-api-access-b45j5\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:31:36.481697 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.481665 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48052ae2-569f-4dd6-8f43-5aeafa311a82" path="/var/lib/kubelet/pods/48052ae2-569f-4dd6-8f43-5aeafa311a82/volumes" Jul 14 18:31:36.979394 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.979358 2569 generic.go:356] "Generic (PLEG): container finished" podID="15de8ed9-a5cb-4d67-8e8e-2fb84accae85" containerID="23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0" exitCode=0 Jul 14 18:31:36.979705 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.979405 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6455c599f7-hq2cg" Jul 14 18:31:36.979705 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.979413 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6455c599f7-hq2cg" event={"ID":"15de8ed9-a5cb-4d67-8e8e-2fb84accae85","Type":"ContainerDied","Data":"23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0"} Jul 14 18:31:36.979705 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.979445 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6455c599f7-hq2cg" event={"ID":"15de8ed9-a5cb-4d67-8e8e-2fb84accae85","Type":"ContainerDied","Data":"a6db9c389eddbcbfa58c2cf787eaa90422858a4a89effa67a94a3e76d783c270"} Jul 14 18:31:36.979705 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.979463 2569 scope.go:117] "RemoveContainer" containerID="23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0" Jul 14 18:31:36.988041 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.988016 2569 scope.go:117] "RemoveContainer" containerID="23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0" Jul 14 18:31:36.988380 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:31:36.988361 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0\": container with ID starting with 23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0 not found: ID does not exist" containerID="23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0" Jul 14 18:31:36.988434 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:36.988389 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0"} err="failed to get container status \"23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0\": rpc error: code = NotFound desc = could not find container \"23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0\": container with ID starting with 23bc3ff7c49719cf878509a8c455bb3a769faa51a06149bb3a617a217c0f52b0 not found: ID does not exist" Jul 14 18:31:37.009192 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:37.009153 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6455c599f7-hq2cg"] Jul 14 18:31:37.015446 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:37.015413 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6455c599f7-hq2cg"] Jul 14 18:31:38.482291 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:31:38.482226 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15de8ed9-a5cb-4d67-8e8e-2fb84accae85" path="/var/lib/kubelet/pods/15de8ed9-a5cb-4d67-8e8e-2fb84accae85/volumes" Jul 14 18:32:19.092900 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:32:19.092815 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:32:45.894677 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:32:45.894634 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:32:51.695389 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:32:51.695355 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:32:59.267811 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:32:59.267778 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:33:04.214574 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:04.214534 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:33:16.995533 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:16.995492 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l"] Jul 14 18:33:16.995971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:16.995818 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15de8ed9-a5cb-4d67-8e8e-2fb84accae85" containerName="authorino" Jul 14 18:33:16.995971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:16.995830 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="15de8ed9-a5cb-4d67-8e8e-2fb84accae85" containerName="authorino" Jul 14 18:33:16.995971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:16.995845 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48052ae2-569f-4dd6-8f43-5aeafa311a82" containerName="authorino" Jul 14 18:33:16.995971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:16.995852 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="48052ae2-569f-4dd6-8f43-5aeafa311a82" containerName="authorino" Jul 14 18:33:16.995971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:16.995917 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="48052ae2-569f-4dd6-8f43-5aeafa311a82" containerName="authorino" Jul 14 18:33:16.995971 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:16.995929 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="15de8ed9-a5cb-4d67-8e8e-2fb84accae85" containerName="authorino" Jul 14 18:33:16.997846 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:16.997828 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.000894 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.000875 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 14 18:33:17.001958 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.001935 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-bqqxv\"" Jul 14 18:33:17.002068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.001995 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 14 18:33:17.002068 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.001996 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 14 18:33:17.015012 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.014984 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l"] Jul 14 18:33:17.026870 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.026842 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.027026 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.026892 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.027026 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.026933 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b569c6ab-9898-4f57-a4e2-94ece72a1a19-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.027026 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.026950 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.027026 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.026973 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.027026 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.026987 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-526ml\" (UniqueName: \"kubernetes.io/projected/b569c6ab-9898-4f57-a4e2-94ece72a1a19-kube-api-access-526ml\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.127850 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.127813 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.128031 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.127869 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.128031 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.127894 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b569c6ab-9898-4f57-a4e2-94ece72a1a19-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.128031 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.127915 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.128031 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.127950 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.128031 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.127974 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-526ml\" (UniqueName: \"kubernetes.io/projected/b569c6ab-9898-4f57-a4e2-94ece72a1a19-kube-api-access-526ml\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.128405 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.128381 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.128458 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.128394 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.128504 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.128458 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.130263 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.130231 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b569c6ab-9898-4f57-a4e2-94ece72a1a19-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.130613 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.130595 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b569c6ab-9898-4f57-a4e2-94ece72a1a19-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.140217 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.140184 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-526ml\" (UniqueName: \"kubernetes.io/projected/b569c6ab-9898-4f57-a4e2-94ece72a1a19-kube-api-access-526ml\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-kg25l\" (UID: \"b569c6ab-9898-4f57-a4e2-94ece72a1a19\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.308019 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.307982 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:17.442668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.442644 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l"] Jul 14 18:33:17.443546 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:33:17.443523 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb569c6ab_9898_4f57_a4e2_94ece72a1a19.slice/crio-b9519a23747c1f23eb3dc6d7389e98dea566b5db56a5312bec7adbf0451e8e96 WatchSource:0}: Error finding container b9519a23747c1f23eb3dc6d7389e98dea566b5db56a5312bec7adbf0451e8e96: Status 404 returned error can't find the container with id b9519a23747c1f23eb3dc6d7389e98dea566b5db56a5312bec7adbf0451e8e96 Jul 14 18:33:17.444828 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:17.444809 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 18:33:18.441912 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:18.441873 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" event={"ID":"b569c6ab-9898-4f57-a4e2-94ece72a1a19","Type":"ContainerStarted","Data":"b9519a23747c1f23eb3dc6d7389e98dea566b5db56a5312bec7adbf0451e8e96"} Jul 14 18:33:18.898768 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:18.898724 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:33:19.449734 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:19.449697 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" event={"ID":"b569c6ab-9898-4f57-a4e2-94ece72a1a19","Type":"ContainerStarted","Data":"d414a7524c083d61c0da84b5cacde0a4682dca0e482daf192d23ab17e07741f2"} Jul 14 18:33:19.478127 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:19.478073 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" podStartSLOduration=2.47074423 podStartE2EDuration="3.478055516s" totalImagesPullingTime="1.007311286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:33:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:33:19.477682738 +0000 UTC m=+693.535350783" watchObservedRunningTime="2026-07-14 18:33:19.478055516 +0000 UTC m=+693.535723565" Jul 14 18:33:23.284337 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.282132 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx"] Jul 14 18:33:23.285814 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.285784 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.289649 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.289628 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 14 18:33:23.301484 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.301453 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx"] Jul 14 18:33:23.380440 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.380401 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.380440 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.380446 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.380686 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.380526 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37b2919-d842-46d2-98ea-b3c464ef6806-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.380686 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.380580 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zwsgz\" (UniqueName: \"kubernetes.io/projected/b37b2919-d842-46d2-98ea-b3c464ef6806-kube-api-access-zwsgz\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.380686 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.380642 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.380686 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.380676 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.481662 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.481623 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zwsgz\" (UniqueName: \"kubernetes.io/projected/b37b2919-d842-46d2-98ea-b3c464ef6806-kube-api-access-zwsgz\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.481847 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.481673 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.481847 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.481694 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.481847 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.481717 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.481847 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.481744 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.481847 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.481799 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37b2919-d842-46d2-98ea-b3c464ef6806-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.482108 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.482089 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.482154 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.482128 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.482200 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.482180 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.483982 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.483962 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b37b2919-d842-46d2-98ea-b3c464ef6806-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.484229 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.484211 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b37b2919-d842-46d2-98ea-b3c464ef6806-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.491715 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.491692 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zwsgz\" (UniqueName: \"kubernetes.io/projected/b37b2919-d842-46d2-98ea-b3c464ef6806-kube-api-access-zwsgz\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx\" (UID: \"b37b2919-d842-46d2-98ea-b3c464ef6806\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.595303 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.595192 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:23.739875 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:23.739847 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx"] Jul 14 18:33:23.741465 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:33:23.741438 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb37b2919_d842_46d2_98ea_b3c464ef6806.slice/crio-a22f8d4fd8e84b2b89c4eb5bf4eb8e45d6916d12ffbe7f27acdb086b8d0a1861 WatchSource:0}: Error finding container a22f8d4fd8e84b2b89c4eb5bf4eb8e45d6916d12ffbe7f27acdb086b8d0a1861: Status 404 returned error can't find the container with id a22f8d4fd8e84b2b89c4eb5bf4eb8e45d6916d12ffbe7f27acdb086b8d0a1861 Jul 14 18:33:24.473688 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:24.473654 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" event={"ID":"b37b2919-d842-46d2-98ea-b3c464ef6806","Type":"ContainerStarted","Data":"f05fc3b27894fca18d15c450a6a952a11f744dd836a1e650d63a0b233e41d2d9"} Jul 14 18:33:24.474072 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:24.473696 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" event={"ID":"b37b2919-d842-46d2-98ea-b3c464ef6806","Type":"ContainerStarted","Data":"a22f8d4fd8e84b2b89c4eb5bf4eb8e45d6916d12ffbe7f27acdb086b8d0a1861"} Jul 14 18:33:24.501682 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:24.501636 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" podStartSLOduration=1.315110491 podStartE2EDuration="1.501617386s" totalImagesPullingTime="186.506895ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:33:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:33:24.500128692 +0000 UTC m=+698.557796742" watchObservedRunningTime="2026-07-14 18:33:24.501617386 +0000 UTC m=+698.559285435" Jul 14 18:33:26.700061 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:26.700026 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:33:27.308246 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:27.308202 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:27.308462 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:27.308301 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:27.320746 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:27.320720 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:27.498423 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:27.498392 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-kg25l" Jul 14 18:33:33.595999 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:33.595953 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:33.596454 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:33.596012 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:33.608319 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:33.608291 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:33:34.536437 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:33:34.536406 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx" Jul 14 18:34:28.591710 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.591672 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-68797955b-nkq9q"] Jul 14 18:34:28.594663 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.594647 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68797955b-nkq9q" Jul 14 18:34:28.598917 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.598886 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-z7rpx\"" Jul 14 18:34:28.599056 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.598917 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 14 18:34:28.604379 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.604357 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68797955b-nkq9q"] Jul 14 18:34:28.650237 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.650207 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/90012bb3-7876-497e-a403-08c133191e2e-tls-cert\") pod \"authorino-68797955b-nkq9q\" (UID: \"90012bb3-7876-497e-a403-08c133191e2e\") " pod="kuadrant-system/authorino-68797955b-nkq9q" Jul 14 18:34:28.650442 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.650279 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gplws\" (UniqueName: \"kubernetes.io/projected/90012bb3-7876-497e-a403-08c133191e2e-kube-api-access-gplws\") pod \"authorino-68797955b-nkq9q\" (UID: \"90012bb3-7876-497e-a403-08c133191e2e\") " pod="kuadrant-system/authorino-68797955b-nkq9q" Jul 14 18:34:28.750976 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.750941 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/90012bb3-7876-497e-a403-08c133191e2e-tls-cert\") pod \"authorino-68797955b-nkq9q\" (UID: \"90012bb3-7876-497e-a403-08c133191e2e\") " pod="kuadrant-system/authorino-68797955b-nkq9q" Jul 14 18:34:28.750976 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.750989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gplws\" (UniqueName: \"kubernetes.io/projected/90012bb3-7876-497e-a403-08c133191e2e-kube-api-access-gplws\") pod \"authorino-68797955b-nkq9q\" (UID: \"90012bb3-7876-497e-a403-08c133191e2e\") " pod="kuadrant-system/authorino-68797955b-nkq9q" Jul 14 18:34:28.753442 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.753417 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/90012bb3-7876-497e-a403-08c133191e2e-tls-cert\") pod \"authorino-68797955b-nkq9q\" (UID: \"90012bb3-7876-497e-a403-08c133191e2e\") " pod="kuadrant-system/authorino-68797955b-nkq9q" Jul 14 18:34:28.760429 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.760398 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gplws\" (UniqueName: \"kubernetes.io/projected/90012bb3-7876-497e-a403-08c133191e2e-kube-api-access-gplws\") pod \"authorino-68797955b-nkq9q\" (UID: \"90012bb3-7876-497e-a403-08c133191e2e\") " pod="kuadrant-system/authorino-68797955b-nkq9q" Jul 14 18:34:28.904896 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:28.904807 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-68797955b-nkq9q" Jul 14 18:34:29.039792 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:29.039762 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-68797955b-nkq9q"] Jul 14 18:34:29.041750 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:34:29.041727 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod90012bb3_7876_497e_a403_08c133191e2e.slice/crio-a66e134919d0da3e42147b9342361bac9b1fb5ae66f71a988954964beb77a4d4 WatchSource:0}: Error finding container a66e134919d0da3e42147b9342361bac9b1fb5ae66f71a988954964beb77a4d4: Status 404 returned error can't find the container with id a66e134919d0da3e42147b9342361bac9b1fb5ae66f71a988954964beb77a4d4 Jul 14 18:34:29.786120 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:29.786078 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68797955b-nkq9q" event={"ID":"90012bb3-7876-497e-a403-08c133191e2e","Type":"ContainerStarted","Data":"603bac30b5a66e1533b584bc23e94b4aae706454a649ac41f6044c99714bd839"} Jul 14 18:34:29.786612 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:29.786134 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-68797955b-nkq9q" event={"ID":"90012bb3-7876-497e-a403-08c133191e2e","Type":"ContainerStarted","Data":"a66e134919d0da3e42147b9342361bac9b1fb5ae66f71a988954964beb77a4d4"} Jul 14 18:34:29.806650 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:29.806589 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-68797955b-nkq9q" podStartSLOduration=1.38114647 podStartE2EDuration="1.80656679s" totalImagesPullingTime="425.42032ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:34:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:34:29.804231817 +0000 UTC m=+763.861899867" watchObservedRunningTime="2026-07-14 18:34:29.80656679 +0000 UTC m=+763.864234839" Jul 14 18:34:55.588966 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.588917 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-67449cbd47-2kmn9"] Jul 14 18:34:55.592607 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.592588 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.607408 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.607378 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67449cbd47-2kmn9"] Jul 14 18:34:55.689959 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.689912 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-oauth-serving-cert\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.689959 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.689953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-serving-cert\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.689959 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.689969 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-service-ca\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.690292 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.690055 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-oauth-config\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.690292 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.690106 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-trusted-ca-bundle\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.690292 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.690126 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbln2\" (UniqueName: \"kubernetes.io/projected/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-kube-api-access-sbln2\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.690292 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.690188 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-config\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.790769 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.790721 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-trusted-ca-bundle\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.790769 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.790771 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbln2\" (UniqueName: \"kubernetes.io/projected/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-kube-api-access-sbln2\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791093 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.790836 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-config\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791093 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.790883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-oauth-serving-cert\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791093 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.790908 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-serving-cert\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791093 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.790931 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-service-ca\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791093 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.790970 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-oauth-config\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791587 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.791560 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-config\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.791670 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-trusted-ca-bundle\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791736 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.791692 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-oauth-serving-cert\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.791736 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.791714 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-service-ca\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.793541 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.793520 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-oauth-config\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.793643 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.793597 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-console-serving-cert\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.800374 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.800354 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbln2\" (UniqueName: \"kubernetes.io/projected/16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f-kube-api-access-sbln2\") pod \"console-67449cbd47-2kmn9\" (UID: \"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f\") " pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:55.901887 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:55.901800 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:34:56.042556 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:56.042528 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-67449cbd47-2kmn9"] Jul 14 18:34:56.045498 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:34:56.045469 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e9f4a2_15bf_46e5_93ed_9ddf19cdd87f.slice/crio-367627133cb73ff83be0ed43ef8dfac9fd01e260497c0720d01082699a83f946 WatchSource:0}: Error finding container 367627133cb73ff83be0ed43ef8dfac9fd01e260497c0720d01082699a83f946: Status 404 returned error can't find the container with id 367627133cb73ff83be0ed43ef8dfac9fd01e260497c0720d01082699a83f946 Jul 14 18:34:56.927182 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:56.927146 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67449cbd47-2kmn9" event={"ID":"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f","Type":"ContainerStarted","Data":"c09a2980df18081a9352269c761b49c02ffd9d676b0663831e29c5d8b7a81d48"} Jul 14 18:34:56.927656 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:56.927200 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-67449cbd47-2kmn9" event={"ID":"16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f","Type":"ContainerStarted","Data":"367627133cb73ff83be0ed43ef8dfac9fd01e260497c0720d01082699a83f946"} Jul 14 18:34:56.946977 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:34:56.946920 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-67449cbd47-2kmn9" podStartSLOduration=1.9469037070000001 podStartE2EDuration="1.946903707s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:34:56.946708956 +0000 UTC m=+791.004377032" watchObservedRunningTime="2026-07-14 18:34:56.946903707 +0000 UTC m=+791.004571755" Jul 14 18:35:01.287801 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:01.287757 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:35:05.902399 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:05.902344 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:35:05.902923 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:05.902691 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:35:05.907279 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:05.907238 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:35:05.981319 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:05.978074 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-67449cbd47-2kmn9" Jul 14 18:35:06.035368 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:06.035333 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6844976579-cwnfs"] Jul 14 18:35:11.524705 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:11.524625 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:35:22.586579 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:22.586545 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:35:31.055521 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.055471 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6844976579-cwnfs" podUID="c50ad9c4-8104-4fd2-8518-822a65f98f73" containerName="console" containerID="cri-o://3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1" gracePeriod=15 Jul 14 18:35:31.186968 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.186937 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:35:31.301481 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.301450 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:35:31.311896 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.311824 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-service-ca\") pod \"c50ad9c4-8104-4fd2-8518-822a65f98f73\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " Jul 14 18:35:31.311896 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.311858 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-oauth-config\") pod \"c50ad9c4-8104-4fd2-8518-822a65f98f73\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " Jul 14 18:35:31.311896 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.311882 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrrhf\" (UniqueName: \"kubernetes.io/projected/c50ad9c4-8104-4fd2-8518-822a65f98f73-kube-api-access-rrrhf\") pod \"c50ad9c4-8104-4fd2-8518-822a65f98f73\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " Jul 14 18:35:31.312106 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.311911 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-config\") pod \"c50ad9c4-8104-4fd2-8518-822a65f98f73\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " Jul 14 18:35:31.312106 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312032 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-serving-cert\") pod \"c50ad9c4-8104-4fd2-8518-822a65f98f73\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " Jul 14 18:35:31.312216 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312123 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-oauth-serving-cert\") pod \"c50ad9c4-8104-4fd2-8518-822a65f98f73\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " Jul 14 18:35:31.312216 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312206 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-trusted-ca-bundle\") pod \"c50ad9c4-8104-4fd2-8518-822a65f98f73\" (UID: \"c50ad9c4-8104-4fd2-8518-822a65f98f73\") " Jul 14 18:35:31.312357 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312225 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-service-ca" (OuterVolumeSpecName: "service-ca") pod "c50ad9c4-8104-4fd2-8518-822a65f98f73" (UID: "c50ad9c4-8104-4fd2-8518-822a65f98f73"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:35:31.312458 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312317 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-config" (OuterVolumeSpecName: "console-config") pod "c50ad9c4-8104-4fd2-8518-822a65f98f73" (UID: "c50ad9c4-8104-4fd2-8518-822a65f98f73"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:35:31.312596 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312468 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "c50ad9c4-8104-4fd2-8518-822a65f98f73" (UID: "c50ad9c4-8104-4fd2-8518-822a65f98f73"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:35:31.312596 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312501 2569 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-service-ca\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:35:31.312596 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312522 2569 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:35:31.312766 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.312651 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "c50ad9c4-8104-4fd2-8518-822a65f98f73" (UID: "c50ad9c4-8104-4fd2-8518-822a65f98f73"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 14 18:35:31.314122 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.314094 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "c50ad9c4-8104-4fd2-8518-822a65f98f73" (UID: "c50ad9c4-8104-4fd2-8518-822a65f98f73"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:35:31.314219 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.314125 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c50ad9c4-8104-4fd2-8518-822a65f98f73-kube-api-access-rrrhf" (OuterVolumeSpecName: "kube-api-access-rrrhf") pod "c50ad9c4-8104-4fd2-8518-822a65f98f73" (UID: "c50ad9c4-8104-4fd2-8518-822a65f98f73"). InnerVolumeSpecName "kube-api-access-rrrhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:35:31.314219 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.314154 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "c50ad9c4-8104-4fd2-8518-822a65f98f73" (UID: "c50ad9c4-8104-4fd2-8518-822a65f98f73"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:35:31.413754 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.413711 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrrhf\" (UniqueName: \"kubernetes.io/projected/c50ad9c4-8104-4fd2-8518-822a65f98f73-kube-api-access-rrrhf\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:35:31.413754 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.413746 2569 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:35:31.413754 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.413755 2569 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-oauth-serving-cert\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:35:31.413754 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.413763 2569 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/c50ad9c4-8104-4fd2-8518-822a65f98f73-trusted-ca-bundle\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:35:31.414038 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:31.413772 2569 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/c50ad9c4-8104-4fd2-8518-822a65f98f73-console-oauth-config\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:35:32.107957 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.107923 2569 generic.go:356] "Generic (PLEG): container finished" podID="c50ad9c4-8104-4fd2-8518-822a65f98f73" containerID="3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1" exitCode=0 Jul 14 18:35:32.108344 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.107983 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6844976579-cwnfs" Jul 14 18:35:32.108344 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.108010 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6844976579-cwnfs" event={"ID":"c50ad9c4-8104-4fd2-8518-822a65f98f73","Type":"ContainerDied","Data":"3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1"} Jul 14 18:35:32.108344 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.108101 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6844976579-cwnfs" event={"ID":"c50ad9c4-8104-4fd2-8518-822a65f98f73","Type":"ContainerDied","Data":"08736449ce4f21bbfa35cebbfb1c10f3fc2df19a67eecfb6a31267f924a34a76"} Jul 14 18:35:32.108344 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.108126 2569 scope.go:117] "RemoveContainer" containerID="3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1" Jul 14 18:35:32.116856 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.116836 2569 scope.go:117] "RemoveContainer" containerID="3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1" Jul 14 18:35:32.117111 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:35:32.117091 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1\": container with ID starting with 3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1 not found: ID does not exist" containerID="3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1" Jul 14 18:35:32.117165 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.117118 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1"} err="failed to get container status \"3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1\": rpc error: code = NotFound desc = could not find container \"3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1\": container with ID starting with 3d8d6ade015dd50ce375cd9b9659251bc3b72d2dad5eef2b088bdfd1c5ef8ba1 not found: ID does not exist" Jul 14 18:35:32.130048 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.130023 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6844976579-cwnfs"] Jul 14 18:35:32.134974 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.134948 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6844976579-cwnfs"] Jul 14 18:35:32.482498 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:32.482424 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c50ad9c4-8104-4fd2-8518-822a65f98f73" path="/var/lib/kubelet/pods/c50ad9c4-8104-4fd2-8518-822a65f98f73/volumes" Jul 14 18:35:42.887656 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:35:42.887624 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:36:10.385413 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:36:10.385365 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:36:22.184526 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:36:22.184492 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:36:57.589424 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:36:57.589387 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:37:07.199450 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:37:07.199415 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:37:16.292045 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:37:16.292008 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:37:27.190649 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:37:27.190609 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:37:39.298751 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:37:39.298717 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:37:50.796333 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:37:50.796303 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:38:01.683981 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:38:01.683938 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:38:11.287385 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:38:11.287292 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:38:37.683792 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:38:37.683752 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:38:48.684186 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:38:48.684150 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:39:10.991711 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:39:10.991679 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:39:23.403745 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:39:23.403695 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:39:39.706196 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:39:39.706113 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:39:49.896830 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:39:49.896793 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:39:59.488975 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:39:59.488936 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:40:09.087983 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:40:09.087947 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:40:18.390416 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:40:18.390381 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:42:04.797825 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:42:04.797789 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:42:14.693862 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:42:14.693804 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:42:25.399666 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:42:25.399629 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:42:38.599992 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:42:38.599887 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:42:48.001858 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:42:48.001821 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:42:58.284174 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:42:58.284136 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:43:07.189488 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:43:07.189450 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:43:17.885196 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:43:17.885156 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:43:29.428207 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:43:29.428174 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:43:36.890648 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:43:36.890608 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:43:45.988844 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:43:45.988808 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:43:55.992736 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:43:55.992699 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:44:13.596466 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:44:13.596373 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:44:22.690197 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:44:22.690165 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:44:34.290026 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:44:34.289993 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:44:43.542107 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:44:43.542070 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:44:59.985338 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:44:59.985302 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:45:11.086564 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:45:11.086532 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:45:19.788264 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:45:19.788213 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:45:30.097089 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:45:30.097052 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:45:39.590171 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:45:39.589959 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:45:48.903942 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:45:48.903898 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:46:02.090184 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:46:02.090145 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:46:11.683878 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:46:11.683836 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:46:21.038923 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:46:21.038889 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:46:33.693706 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:46:33.693669 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:46:45.400599 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:46:45.400561 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:46:54.284054 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:46:54.284010 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:47:06.491455 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:47:06.491420 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:47:18.294340 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:47:18.294220 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:47:29.191873 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:47:29.191833 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:47:41.692707 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:47:41.692668 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:47:50.681565 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:47:50.681532 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:47:59.886717 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:47:59.886681 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:48:10.486991 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:48:10.486956 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:49:11.490391 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:49:11.490310 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:49:27.780848 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:49:27.780818 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:50:33.385164 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:50:33.385128 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:50:43.993506 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:50:43.993475 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:51:47.390918 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:51:47.390880 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:52:04.284581 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:52:04.284543 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:53:19.789829 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:53:19.789794 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:54:57.817888 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:54:57.817799 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:55:05.786605 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:55:05.786568 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:55:14.885023 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:55:14.884988 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:55:25.708693 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:55:25.708649 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:55:42.091994 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:55:42.091956 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:56:26.393035 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:26.392999 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:56:33.875213 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.875178 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb"] Jul 14 18:56:33.875682 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.875509 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c50ad9c4-8104-4fd2-8518-822a65f98f73" containerName="console" Jul 14 18:56:33.875682 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.875522 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c50ad9c4-8104-4fd2-8518-822a65f98f73" containerName="console" Jul 14 18:56:33.875682 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.875584 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c50ad9c4-8104-4fd2-8518-822a65f98f73" containerName="console" Jul 14 18:56:33.878587 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.878560 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:33.881940 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.881914 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-99a89b27\"/\"kube-root-ca.crt\"" Jul 14 18:56:33.882662 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.882640 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-99a89b27\"/\"e2e-lifecycle-model-99a89b27-kserve-self-signed-certs\"" Jul 14 18:56:33.882795 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.882687 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-99a89b27\"/\"openshift-service-ca.crt\"" Jul 14 18:56:33.883130 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.883116 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-99a89b27\"/\"default-dockercfg-xgkgm\"" Jul 14 18:56:33.889279 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.889236 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb"] Jul 14 18:56:33.959631 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.959596 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-model-cache\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:33.959631 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.959635 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tmp-dir\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:33.959889 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.959668 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnp62\" (UniqueName: \"kubernetes.io/projected/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-kube-api-access-jnp62\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:33.959889 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.959739 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-dshm\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:33.959889 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.959829 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-home\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:33.959889 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:33.959864 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tls-certs\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.060954 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.060910 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-model-cache\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.060954 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.060956 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tmp-dir\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.061209 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.060989 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnp62\" (UniqueName: \"kubernetes.io/projected/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-kube-api-access-jnp62\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.061209 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.061030 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-dshm\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.061209 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.061084 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-home\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.061209 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.061106 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tls-certs\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.061463 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.061428 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-model-cache\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.061523 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.061493 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tmp-dir\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.061523 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.061508 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-home\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.063338 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.063278 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-dshm\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.063561 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.063543 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tls-certs\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.071609 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.071583 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnp62\" (UniqueName: \"kubernetes.io/projected/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-kube-api-access-jnp62\") pod \"e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.189508 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.189409 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:34.324288 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:34.324247 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb"] Jul 14 18:56:34.325825 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:56:34.325796 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc9cdd857_d133_4ece_a1d8_95fcfdb255e8.slice/crio-6399099f7f3903fd2231aafe2808043747c1d8a04c7c99e4fa6cc90c1518d11c WatchSource:0}: Error finding container 6399099f7f3903fd2231aafe2808043747c1d8a04c7c99e4fa6cc90c1518d11c: Status 404 returned error can't find the container with id 6399099f7f3903fd2231aafe2808043747c1d8a04c7c99e4fa6cc90c1518d11c Jul 14 18:56:35.217678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:35.217636 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" event={"ID":"c9cdd857-d133-4ece-a1d8-95fcfdb255e8","Type":"ContainerStarted","Data":"126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526"} Jul 14 18:56:35.217678 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:35.217684 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" event={"ID":"c9cdd857-d133-4ece-a1d8-95fcfdb255e8","Type":"ContainerStarted","Data":"6399099f7f3903fd2231aafe2808043747c1d8a04c7c99e4fa6cc90c1518d11c"} Jul 14 18:56:35.236584 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:35.236530 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" podStartSLOduration=2.236513056 podStartE2EDuration="2.236513056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:56:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:56:35.235918861 +0000 UTC m=+2089.293586931" watchObservedRunningTime="2026-07-14 18:56:35.236513056 +0000 UTC m=+2089.294181105" Jul 14 18:56:40.382341 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:40.382300 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:56:44.189872 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:44.189833 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:44.189872 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:44.189883 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:44.202572 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:44.202544 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:44.273535 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:44.273499 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:56:49.587438 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:56:49.587396 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:57:06.691651 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:06.691599 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:57:12.540387 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:12.540349 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb"] Jul 14 18:57:27.561160 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.561118 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" podUID="c9cdd857-d133-4ece-a1d8-95fcfdb255e8" containerName="main" containerID="cri-o://126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526" gracePeriod=45 Jul 14 18:57:27.816244 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.816176 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:57:27.936076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936040 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tls-certs\") pod \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " Jul 14 18:57:27.936076 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936084 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tmp-dir\") pod \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " Jul 14 18:57:27.936349 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936142 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jnp62\" (UniqueName: \"kubernetes.io/projected/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-kube-api-access-jnp62\") pod \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " Jul 14 18:57:27.936349 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936175 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-dshm\") pod \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " Jul 14 18:57:27.936349 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936200 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-model-cache\") pod \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " Jul 14 18:57:27.936349 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936222 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-home\") pod \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\" (UID: \"c9cdd857-d133-4ece-a1d8-95fcfdb255e8\") " Jul 14 18:57:27.936562 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936462 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c9cdd857-d133-4ece-a1d8-95fcfdb255e8" (UID: "c9cdd857-d133-4ece-a1d8-95fcfdb255e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:57:27.936562 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936508 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c9cdd857-d133-4ece-a1d8-95fcfdb255e8" (UID: "c9cdd857-d133-4ece-a1d8-95fcfdb255e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:57:27.936668 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.936591 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-home" (OuterVolumeSpecName: "home") pod "c9cdd857-d133-4ece-a1d8-95fcfdb255e8" (UID: "c9cdd857-d133-4ece-a1d8-95fcfdb255e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:57:27.938400 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.938377 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-dshm" (OuterVolumeSpecName: "dshm") pod "c9cdd857-d133-4ece-a1d8-95fcfdb255e8" (UID: "c9cdd857-d133-4ece-a1d8-95fcfdb255e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 18:57:27.938470 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.938402 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c9cdd857-d133-4ece-a1d8-95fcfdb255e8" (UID: "c9cdd857-d133-4ece-a1d8-95fcfdb255e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 18:57:27.938508 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:27.938495 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-kube-api-access-jnp62" (OuterVolumeSpecName: "kube-api-access-jnp62") pod "c9cdd857-d133-4ece-a1d8-95fcfdb255e8" (UID: "c9cdd857-d133-4ece-a1d8-95fcfdb255e8"). InnerVolumeSpecName "kube-api-access-jnp62". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 18:57:28.037774 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.037733 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:57:28.037774 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.037764 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:57:28.037774 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.037775 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:57:28.037774 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.037783 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:57:28.038041 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.037813 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:57:28.038041 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.037822 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnp62\" (UniqueName: \"kubernetes.io/projected/c9cdd857-d133-4ece-a1d8-95fcfdb255e8-kube-api-access-jnp62\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 18:57:28.478208 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.478174 2569 generic.go:356] "Generic (PLEG): container finished" podID="c9cdd857-d133-4ece-a1d8-95fcfdb255e8" containerID="126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526" exitCode=0 Jul 14 18:57:28.478518 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.478289 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" Jul 14 18:57:28.483608 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.483566 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" event={"ID":"c9cdd857-d133-4ece-a1d8-95fcfdb255e8","Type":"ContainerDied","Data":"126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526"} Jul 14 18:57:28.483743 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.483616 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb" event={"ID":"c9cdd857-d133-4ece-a1d8-95fcfdb255e8","Type":"ContainerDied","Data":"6399099f7f3903fd2231aafe2808043747c1d8a04c7c99e4fa6cc90c1518d11c"} Jul 14 18:57:28.483743 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.483641 2569 scope.go:117] "RemoveContainer" containerID="126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526" Jul 14 18:57:28.493436 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.493410 2569 scope.go:117] "RemoveContainer" containerID="126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526" Jul 14 18:57:28.493744 ip-10-0-130-79 kubenswrapper[2569]: E0714 18:57:28.493724 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526\": container with ID starting with 126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526 not found: ID does not exist" containerID="126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526" Jul 14 18:57:28.493802 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.493753 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526"} err="failed to get container status \"126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526\": rpc error: code = NotFound desc = could not find container \"126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526\": container with ID starting with 126be83a3e5684e40528910ad9c51be6f71b794c73435b47b0fe64ce768d8526 not found: ID does not exist" Jul 14 18:57:28.503625 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.503597 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb"] Jul 14 18:57:28.507012 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:28.506990 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-99a89b27/e2e-lifecycle-model-99a89b27-kserve-656788f5cf-zjsqb"] Jul 14 18:57:30.481735 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:57:30.481697 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c9cdd857-d133-4ece-a1d8-95fcfdb255e8" path="/var/lib/kubelet/pods/c9cdd857-d133-4ece-a1d8-95fcfdb255e8/volumes" Jul 14 18:58:04.290692 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:58:04.290653 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:58:18.997553 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:58:18.997519 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:59:11.787241 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:11.787158 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:59:23.468709 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.468674 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt"] Jul 14 18:59:23.469134 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.469065 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c9cdd857-d133-4ece-a1d8-95fcfdb255e8" containerName="main" Jul 14 18:59:23.469134 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.469078 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="c9cdd857-d133-4ece-a1d8-95fcfdb255e8" containerName="main" Jul 14 18:59:23.469134 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.469134 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="c9cdd857-d133-4ece-a1d8-95fcfdb255e8" containerName="main" Jul 14 18:59:23.472066 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.472048 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.477853 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.477831 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0895b5\"/\"kube-root-ca.crt\"" Jul 14 18:59:23.478001 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.477834 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0895b5\"/\"default-dockercfg-74xst\"" Jul 14 18:59:23.478001 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.477834 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0895b5\"/\"test-model-0895b5-kserve-self-signed-certs\"" Jul 14 18:59:23.478001 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.477910 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0895b5\"/\"openshift-service-ca.crt\"" Jul 14 18:59:23.485321 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.485296 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt"] Jul 14 18:59:23.523953 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.523910 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-home\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.523953 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.523960 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-dshm\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.524191 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.524001 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tls-certs\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.524191 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.524033 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-model-cache\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.524191 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.524074 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gpcb\" (UniqueName: \"kubernetes.io/projected/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-kube-api-access-7gpcb\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.524191 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.524129 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tmp-dir\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.625310 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625273 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-home\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.625310 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625318 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-dshm\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.625577 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625337 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tls-certs\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.625577 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625456 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-model-cache\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.625577 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625535 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gpcb\" (UniqueName: \"kubernetes.io/projected/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-kube-api-access-7gpcb\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.625734 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625583 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tmp-dir\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.625793 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625734 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-home\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.626056 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625834 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-model-cache\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.626056 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.625912 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tmp-dir\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.627718 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.627699 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-dshm\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.627855 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.627838 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tls-certs\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.636656 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.636629 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gpcb\" (UniqueName: \"kubernetes.io/projected/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-kube-api-access-7gpcb\") pod \"test-model-0895b5-kserve-66b5989f95-75pxt\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.780910 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.780859 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:23.917775 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:23.917748 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt"] Jul 14 18:59:23.919773 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:59:23.919747 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b95be6b_b77e_4a67_ae1c_6add01f92e9b.slice/crio-f444c1acc6f655709a01413125299e7aa858f27f31a3d11f46af0e2c72fcb1aa WatchSource:0}: Error finding container f444c1acc6f655709a01413125299e7aa858f27f31a3d11f46af0e2c72fcb1aa: Status 404 returned error can't find the container with id f444c1acc6f655709a01413125299e7aa858f27f31a3d11f46af0e2c72fcb1aa Jul 14 18:59:24.030390 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:24.030358 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" event={"ID":"1b95be6b-b77e-4a67-ae1c-6add01f92e9b","Type":"ContainerStarted","Data":"2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a"} Jul 14 18:59:24.030571 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:24.030402 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" event={"ID":"1b95be6b-b77e-4a67-ae1c-6add01f92e9b","Type":"ContainerStarted","Data":"f444c1acc6f655709a01413125299e7aa858f27f31a3d11f46af0e2c72fcb1aa"} Jul 14 18:59:24.053422 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:24.053322 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" podStartSLOduration=1.053304669 podStartE2EDuration="1.053304669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:24.051969725 +0000 UTC m=+2258.109637793" watchObservedRunningTime="2026-07-14 18:59:24.053304669 +0000 UTC m=+2258.110972717" Jul 14 18:59:27.189638 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:27.189608 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:59:33.781398 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:33.781362 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:33.782025 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:33.781419 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:33.793815 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:33.793777 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:34.091511 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:34.091417 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 18:59:42.496407 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:42.496373 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 18:59:44.482686 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.482660 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c"] Jul 14 18:59:44.486010 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.485991 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.489001 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.488976 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-08413e\"/\"kube-root-ca.crt\"" Jul 14 18:59:44.490120 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.490083 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-08413e\"/\"openshift-service-ca.crt\"" Jul 14 18:59:44.490120 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.490116 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-08413e\"/\"default-dockercfg-2w9d8\"" Jul 14 18:59:44.490413 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.490399 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-08413e\"/\"test-model-08413e-kserve-self-signed-certs\"" Jul 14 18:59:44.499097 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.499065 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c"] Jul 14 18:59:44.610942 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.610906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df9c014-f909-4762-bccb-025c8f6e0a86-tls-certs\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.611147 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.610953 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbg75\" (UniqueName: \"kubernetes.io/projected/2df9c014-f909-4762-bccb-025c8f6e0a86-kube-api-access-tbg75\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.611147 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.611017 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-dshm\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.611147 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.611138 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-tmp-dir\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.611337 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.611169 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-model-cache\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.611337 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.611197 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-home\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712193 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712159 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbg75\" (UniqueName: \"kubernetes.io/projected/2df9c014-f909-4762-bccb-025c8f6e0a86-kube-api-access-tbg75\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712193 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712200 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-dshm\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712547 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712248 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-tmp-dir\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712547 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712283 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-model-cache\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712547 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712298 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-home\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712547 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712327 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df9c014-f909-4762-bccb-025c8f6e0a86-tls-certs\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712777 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712673 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-tmp-dir\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712777 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712745 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-home\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.712867 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.712784 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-model-cache\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.714571 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.714552 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-dshm\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.714820 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.714799 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df9c014-f909-4762-bccb-025c8f6e0a86-tls-certs\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.721069 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.721046 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbg75\" (UniqueName: \"kubernetes.io/projected/2df9c014-f909-4762-bccb-025c8f6e0a86-kube-api-access-tbg75\") pod \"test-model-08413e-kserve-78dd4bc499-sc67c\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.796240 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.796206 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:44.931118 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:44.931090 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c"] Jul 14 18:59:44.932402 ip-10-0-130-79 kubenswrapper[2569]: W0714 18:59:44.932370 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2df9c014_f909_4762_bccb_025c8f6e0a86.slice/crio-10bb3bbd0c93b8c3e2a30dc183cbd7b7607c3b7aa6eb69087310d7da85fd84ee WatchSource:0}: Error finding container 10bb3bbd0c93b8c3e2a30dc183cbd7b7607c3b7aa6eb69087310d7da85fd84ee: Status 404 returned error can't find the container with id 10bb3bbd0c93b8c3e2a30dc183cbd7b7607c3b7aa6eb69087310d7da85fd84ee Jul 14 18:59:45.136370 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:45.136235 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" event={"ID":"2df9c014-f909-4762-bccb-025c8f6e0a86","Type":"ContainerStarted","Data":"30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6"} Jul 14 18:59:45.136370 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:45.136311 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" event={"ID":"2df9c014-f909-4762-bccb-025c8f6e0a86","Type":"ContainerStarted","Data":"10bb3bbd0c93b8c3e2a30dc183cbd7b7607c3b7aa6eb69087310d7da85fd84ee"} Jul 14 18:59:45.155500 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:45.155444 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" podStartSLOduration=1.155427912 podStartE2EDuration="1.155427912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 18:59:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 18:59:45.153003276 +0000 UTC m=+2279.210671365" watchObservedRunningTime="2026-07-14 18:59:45.155427912 +0000 UTC m=+2279.213095961" Jul 14 18:59:54.796429 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:54.796384 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:54.796429 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:54.796431 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:54.808977 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:54.808947 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:55.195984 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:55.195900 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 18:59:58.195092 ip-10-0-130-79 kubenswrapper[2569]: I0714 18:59:58.195060 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:00:06.356698 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:06.356660 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt"] Jul 14 19:00:07.295608 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:07.295578 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c"] Jul 14 19:00:11.492266 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:11.492216 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:00:21.376446 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.376405 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" podUID="1b95be6b-b77e-4a67-ae1c-6add01f92e9b" containerName="main" containerID="cri-o://2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a" gracePeriod=45 Jul 14 19:00:21.622315 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.622291 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 19:00:21.733132 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733027 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-dshm\") pod \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " Jul 14 19:00:21.733132 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733084 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-model-cache\") pod \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " Jul 14 19:00:21.733132 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733109 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7gpcb\" (UniqueName: \"kubernetes.io/projected/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-kube-api-access-7gpcb\") pod \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " Jul 14 19:00:21.733446 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733137 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-home\") pod \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " Jul 14 19:00:21.733446 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733157 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tmp-dir\") pod \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " Jul 14 19:00:21.733446 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733245 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tls-certs\") pod \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\" (UID: \"1b95be6b-b77e-4a67-ae1c-6add01f92e9b\") " Jul 14 19:00:21.733573 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733459 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "1b95be6b-b77e-4a67-ae1c-6add01f92e9b" (UID: "1b95be6b-b77e-4a67-ae1c-6add01f92e9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:21.733573 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733468 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-home" (OuterVolumeSpecName: "home") pod "1b95be6b-b77e-4a67-ae1c-6add01f92e9b" (UID: "1b95be6b-b77e-4a67-ae1c-6add01f92e9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:21.733573 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733480 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1b95be6b-b77e-4a67-ae1c-6add01f92e9b" (UID: "1b95be6b-b77e-4a67-ae1c-6add01f92e9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:21.733684 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733610 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:21.733684 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733627 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:21.733684 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.733638 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:21.735409 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.735384 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-dshm" (OuterVolumeSpecName: "dshm") pod "1b95be6b-b77e-4a67-ae1c-6add01f92e9b" (UID: "1b95be6b-b77e-4a67-ae1c-6add01f92e9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:21.735513 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.735450 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1b95be6b-b77e-4a67-ae1c-6add01f92e9b" (UID: "1b95be6b-b77e-4a67-ae1c-6add01f92e9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:00:21.735513 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.735458 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-kube-api-access-7gpcb" (OuterVolumeSpecName: "kube-api-access-7gpcb") pod "1b95be6b-b77e-4a67-ae1c-6add01f92e9b" (UID: "1b95be6b-b77e-4a67-ae1c-6add01f92e9b"). InnerVolumeSpecName "kube-api-access-7gpcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:00:21.834197 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.834167 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:21.834197 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.834191 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:21.834197 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:21.834202 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gpcb\" (UniqueName: \"kubernetes.io/projected/1b95be6b-b77e-4a67-ae1c-6add01f92e9b-kube-api-access-7gpcb\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:22.316246 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.316194 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" podUID="2df9c014-f909-4762-bccb-025c8f6e0a86" containerName="main" containerID="cri-o://30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6" gracePeriod=45 Jul 14 19:00:22.318291 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.318268 2569 generic.go:356] "Generic (PLEG): container finished" podID="1b95be6b-b77e-4a67-ae1c-6add01f92e9b" containerID="2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a" exitCode=0 Jul 14 19:00:22.318400 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.318326 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" Jul 14 19:00:22.318400 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.318329 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" event={"ID":"1b95be6b-b77e-4a67-ae1c-6add01f92e9b","Type":"ContainerDied","Data":"2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a"} Jul 14 19:00:22.318506 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.318439 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt" event={"ID":"1b95be6b-b77e-4a67-ae1c-6add01f92e9b","Type":"ContainerDied","Data":"f444c1acc6f655709a01413125299e7aa858f27f31a3d11f46af0e2c72fcb1aa"} Jul 14 19:00:22.318506 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.318471 2569 scope.go:117] "RemoveContainer" containerID="2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a" Jul 14 19:00:22.328214 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.328197 2569 scope.go:117] "RemoveContainer" containerID="2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a" Jul 14 19:00:22.328506 ip-10-0-130-79 kubenswrapper[2569]: E0714 19:00:22.328487 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a\": container with ID starting with 2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a not found: ID does not exist" containerID="2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a" Jul 14 19:00:22.328577 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.328518 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a"} err="failed to get container status \"2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a\": rpc error: code = NotFound desc = could not find container \"2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a\": container with ID starting with 2d2c1ec593f601831124b0ba1614c85474df6e96ca1ee91d28c5ed873855bb9a not found: ID does not exist" Jul 14 19:00:22.383057 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.383025 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt"] Jul 14 19:00:22.386581 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.386557 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0895b5/test-model-0895b5-kserve-66b5989f95-75pxt"] Jul 14 19:00:22.482439 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.482406 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b95be6b-b77e-4a67-ae1c-6add01f92e9b" path="/var/lib/kubelet/pods/1b95be6b-b77e-4a67-ae1c-6add01f92e9b/volumes" Jul 14 19:00:22.560211 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.560187 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 19:00:22.593368 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.593291 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:00:22.641098 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641064 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-home\") pod \"2df9c014-f909-4762-bccb-025c8f6e0a86\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " Jul 14 19:00:22.641284 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641127 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbg75\" (UniqueName: \"kubernetes.io/projected/2df9c014-f909-4762-bccb-025c8f6e0a86-kube-api-access-tbg75\") pod \"2df9c014-f909-4762-bccb-025c8f6e0a86\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " Jul 14 19:00:22.641284 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641151 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-model-cache\") pod \"2df9c014-f909-4762-bccb-025c8f6e0a86\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " Jul 14 19:00:22.641284 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641170 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-tmp-dir\") pod \"2df9c014-f909-4762-bccb-025c8f6e0a86\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " Jul 14 19:00:22.641284 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641202 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-dshm\") pod \"2df9c014-f909-4762-bccb-025c8f6e0a86\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " Jul 14 19:00:22.641512 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641283 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df9c014-f909-4762-bccb-025c8f6e0a86-tls-certs\") pod \"2df9c014-f909-4762-bccb-025c8f6e0a86\" (UID: \"2df9c014-f909-4762-bccb-025c8f6e0a86\") " Jul 14 19:00:22.641512 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641381 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-home" (OuterVolumeSpecName: "home") pod "2df9c014-f909-4762-bccb-025c8f6e0a86" (UID: "2df9c014-f909-4762-bccb-025c8f6e0a86"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:22.641618 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641513 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-model-cache" (OuterVolumeSpecName: "model-cache") pod "2df9c014-f909-4762-bccb-025c8f6e0a86" (UID: "2df9c014-f909-4762-bccb-025c8f6e0a86"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:22.641618 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641528 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2df9c014-f909-4762-bccb-025c8f6e0a86" (UID: "2df9c014-f909-4762-bccb-025c8f6e0a86"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:22.641737 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641650 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:22.641737 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641663 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:22.641737 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.641672 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:22.643571 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.643547 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-dshm" (OuterVolumeSpecName: "dshm") pod "2df9c014-f909-4762-bccb-025c8f6e0a86" (UID: "2df9c014-f909-4762-bccb-025c8f6e0a86"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:00:22.643862 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.643845 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2df9c014-f909-4762-bccb-025c8f6e0a86-kube-api-access-tbg75" (OuterVolumeSpecName: "kube-api-access-tbg75") pod "2df9c014-f909-4762-bccb-025c8f6e0a86" (UID: "2df9c014-f909-4762-bccb-025c8f6e0a86"). InnerVolumeSpecName "kube-api-access-tbg75". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:00:22.643934 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.643861 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2df9c014-f909-4762-bccb-025c8f6e0a86-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2df9c014-f909-4762-bccb-025c8f6e0a86" (UID: "2df9c014-f909-4762-bccb-025c8f6e0a86"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:00:22.742870 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.742839 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbg75\" (UniqueName: \"kubernetes.io/projected/2df9c014-f909-4762-bccb-025c8f6e0a86-kube-api-access-tbg75\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:22.742870 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.742865 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2df9c014-f909-4762-bccb-025c8f6e0a86-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:22.742870 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:22.742875 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2df9c014-f909-4762-bccb-025c8f6e0a86-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:00:23.324995 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.324963 2569 generic.go:356] "Generic (PLEG): container finished" podID="2df9c014-f909-4762-bccb-025c8f6e0a86" containerID="30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6" exitCode=0 Jul 14 19:00:23.325184 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.325007 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" Jul 14 19:00:23.325184 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.325046 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" event={"ID":"2df9c014-f909-4762-bccb-025c8f6e0a86","Type":"ContainerDied","Data":"30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6"} Jul 14 19:00:23.325184 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.325091 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c" event={"ID":"2df9c014-f909-4762-bccb-025c8f6e0a86","Type":"ContainerDied","Data":"10bb3bbd0c93b8c3e2a30dc183cbd7b7607c3b7aa6eb69087310d7da85fd84ee"} Jul 14 19:00:23.325184 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.325107 2569 scope.go:117] "RemoveContainer" containerID="30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6" Jul 14 19:00:23.333684 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.333664 2569 scope.go:117] "RemoveContainer" containerID="30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6" Jul 14 19:00:23.333982 ip-10-0-130-79 kubenswrapper[2569]: E0714 19:00:23.333961 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6\": container with ID starting with 30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6 not found: ID does not exist" containerID="30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6" Jul 14 19:00:23.334035 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.333989 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6"} err="failed to get container status \"30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6\": rpc error: code = NotFound desc = could not find container \"30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6\": container with ID starting with 30e9502a777c8dab4acc9649bd2e610fd245e6531e109ee6c960a10ef02768e6 not found: ID does not exist" Jul 14 19:00:23.348325 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.348296 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c"] Jul 14 19:00:23.352803 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:23.352779 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-08413e/test-model-08413e-kserve-78dd4bc499-sc67c"] Jul 14 19:00:24.482618 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:24.482586 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2df9c014-f909-4762-bccb-025c8f6e0a86" path="/var/lib/kubelet/pods/2df9c014-f909-4762-bccb-025c8f6e0a86/volumes" Jul 14 19:00:47.686620 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:00:47.686570 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:01:09.489083 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:01:09.489044 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:01:27.290699 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:01:27.290662 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:01:38.903941 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:01:38.903905 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:02:04.396465 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:04.396430 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:02:25.793356 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:25.793313 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:02:41.801779 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:41.801744 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:02:45.873473 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.873439 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65"] Jul 14 19:02:45.873842 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.873758 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2df9c014-f909-4762-bccb-025c8f6e0a86" containerName="main" Jul 14 19:02:45.873842 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.873769 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="2df9c014-f909-4762-bccb-025c8f6e0a86" containerName="main" Jul 14 19:02:45.873842 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.873790 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b95be6b-b77e-4a67-ae1c-6add01f92e9b" containerName="main" Jul 14 19:02:45.873842 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.873796 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b95be6b-b77e-4a67-ae1c-6add01f92e9b" containerName="main" Jul 14 19:02:45.873842 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.873844 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b95be6b-b77e-4a67-ae1c-6add01f92e9b" containerName="main" Jul 14 19:02:45.874004 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.873851 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="2df9c014-f909-4762-bccb-025c8f6e0a86" containerName="main" Jul 14 19:02:45.876756 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.876739 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:45.879772 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.879748 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-52ef23\"/\"default-dockercfg-c8d49\"" Jul 14 19:02:45.881106 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.881083 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-52ef23\"/\"openshift-service-ca.crt\"" Jul 14 19:02:45.881246 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.881138 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-52ef23\"/\"kube-root-ca.crt\"" Jul 14 19:02:45.881246 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.881205 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-52ef23\"/\"auth-test-model-52ef23-kserve-self-signed-certs\"" Jul 14 19:02:45.887783 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.887761 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65"] Jul 14 19:02:45.990023 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.989985 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610d0b3a-497d-4a9a-a732-cd306876e8b9-tls-certs\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:45.990023 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.990021 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-tmp-dir\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:45.990238 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.990047 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-model-cache\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:45.990238 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.990122 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-dshm\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:45.990238 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.990155 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-home\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:45.990238 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:45.990173 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k975c\" (UniqueName: \"kubernetes.io/projected/610d0b3a-497d-4a9a-a732-cd306876e8b9-kube-api-access-k975c\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.091605 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.091561 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-model-cache\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.091605 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.091608 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-dshm\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.091866 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.091644 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-home\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.091866 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.091665 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k975c\" (UniqueName: \"kubernetes.io/projected/610d0b3a-497d-4a9a-a732-cd306876e8b9-kube-api-access-k975c\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.091866 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.091766 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610d0b3a-497d-4a9a-a732-cd306876e8b9-tls-certs\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.091866 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.091791 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-tmp-dir\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.092080 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.092037 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-model-cache\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.092080 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.092059 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-home\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.092189 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.092163 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-tmp-dir\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.093985 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.093966 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-dshm\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.094286 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.094246 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610d0b3a-497d-4a9a-a732-cd306876e8b9-tls-certs\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.100148 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.100125 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k975c\" (UniqueName: \"kubernetes.io/projected/610d0b3a-497d-4a9a-a732-cd306876e8b9-kube-api-access-k975c\") pod \"auth-test-model-52ef23-kserve-56cc7d88d7-zth65\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.186077 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.185981 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:46.320265 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:46.320228 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65"] Jul 14 19:02:46.322458 ip-10-0-130-79 kubenswrapper[2569]: W0714 19:02:46.322431 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod610d0b3a_497d_4a9a_a732_cd306876e8b9.slice/crio-579c4256bc5d02e63aa1a90eb89303ee855fdcb25e870745b7f8f0a506ed20ab WatchSource:0}: Error finding container 579c4256bc5d02e63aa1a90eb89303ee855fdcb25e870745b7f8f0a506ed20ab: Status 404 returned error can't find the container with id 579c4256bc5d02e63aa1a90eb89303ee855fdcb25e870745b7f8f0a506ed20ab Jul 14 19:02:47.004190 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:47.004147 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" event={"ID":"610d0b3a-497d-4a9a-a732-cd306876e8b9","Type":"ContainerStarted","Data":"1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46"} Jul 14 19:02:47.004640 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:47.004199 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" event={"ID":"610d0b3a-497d-4a9a-a732-cd306876e8b9","Type":"ContainerStarted","Data":"579c4256bc5d02e63aa1a90eb89303ee855fdcb25e870745b7f8f0a506ed20ab"} Jul 14 19:02:47.026384 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:47.026340 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" podStartSLOduration=2.026324197 podStartE2EDuration="2.026324197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:02:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:02:47.024220254 +0000 UTC m=+2461.081888325" watchObservedRunningTime="2026-07-14 19:02:47.026324197 +0000 UTC m=+2461.083992243" Jul 14 19:02:51.335034 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:51.334993 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:02:56.187168 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:56.187130 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:56.187622 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:56.187182 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:56.199859 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:56.199829 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:02:57.064126 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:02:57.064096 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:03:09.981920 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:09.981881 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf"] Jul 14 19:03:09.985694 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:09.985667 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:09.988636 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:09.988601 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"default-dockercfg-4hckz\"" Jul 14 19:03:09.989369 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:09.989341 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf"] Jul 14 19:03:09.989765 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:09.989746 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"kube-root-ca.crt\"" Jul 14 19:03:09.989899 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:09.989815 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"openshift-service-ca.crt\"" Jul 14 19:03:09.990009 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:09.989829 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"auth-test-model-10fa01-kserve-self-signed-certs\"" Jul 14 19:03:10.106080 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.106036 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-home\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.106316 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.106104 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-dshm\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.106316 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.106180 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-tmp-dir\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.106316 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.106238 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f750c968-7622-41b1-bf0c-a51b49aad79e-tls-certs\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.106469 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.106329 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-model-cache\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.106469 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.106354 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xdqq\" (UniqueName: \"kubernetes.io/projected/f750c968-7622-41b1-bf0c-a51b49aad79e-kube-api-access-2xdqq\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207011 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.206970 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-dshm\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207209 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.207033 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-tmp-dir\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207209 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.207082 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f750c968-7622-41b1-bf0c-a51b49aad79e-tls-certs\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207209 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.207134 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-model-cache\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207209 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.207160 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xdqq\" (UniqueName: \"kubernetes.io/projected/f750c968-7622-41b1-bf0c-a51b49aad79e-kube-api-access-2xdqq\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207209 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.207196 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-home\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207548 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.207523 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-tmp-dir\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207618 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.207567 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-home\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.207618 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.207589 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-model-cache\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.209413 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.209387 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-dshm\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.209692 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.209674 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f750c968-7622-41b1-bf0c-a51b49aad79e-tls-certs\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.222180 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.222147 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xdqq\" (UniqueName: \"kubernetes.io/projected/f750c968-7622-41b1-bf0c-a51b49aad79e-kube-api-access-2xdqq\") pod \"auth-test-model-10fa01-kserve-78b676ddf5-6lbdf\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.298035 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.297999 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:10.441223 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:10.441193 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf"] Jul 14 19:03:10.442818 ip-10-0-130-79 kubenswrapper[2569]: W0714 19:03:10.442793 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf750c968_7622_41b1_bf0c_a51b49aad79e.slice/crio-43ad50de5f8e514c742ef41af10314dd8d5ce49195549fe0a38b4e947ab4f738 WatchSource:0}: Error finding container 43ad50de5f8e514c742ef41af10314dd8d5ce49195549fe0a38b4e947ab4f738: Status 404 returned error can't find the container with id 43ad50de5f8e514c742ef41af10314dd8d5ce49195549fe0a38b4e947ab4f738 Jul 14 19:03:11.121710 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:11.121667 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" event={"ID":"f750c968-7622-41b1-bf0c-a51b49aad79e","Type":"ContainerStarted","Data":"a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd"} Jul 14 19:03:11.122122 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:11.121720 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" event={"ID":"f750c968-7622-41b1-bf0c-a51b49aad79e","Type":"ContainerStarted","Data":"43ad50de5f8e514c742ef41af10314dd8d5ce49195549fe0a38b4e947ab4f738"} Jul 14 19:03:11.150054 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:11.150006 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" podStartSLOduration=2.1499876159999998 podStartE2EDuration="2.149987616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:03:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:03:11.148584821 +0000 UTC m=+2485.206252881" watchObservedRunningTime="2026-07-14 19:03:11.149987616 +0000 UTC m=+2485.207655712" Jul 14 19:03:13.491808 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:13.491776 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:03:20.298436 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:20.298398 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:20.298991 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:20.298622 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:20.311296 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:20.311248 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:21.182050 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:21.182020 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:03:31.322655 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:31.322623 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:03:49.387855 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:03:49.387771 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:04:04.289877 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:04:04.289844 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:04:19.515813 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:04:19.515773 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:04:35.188289 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:04:35.188236 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:04:49.216881 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:04:49.216843 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:05:07.417150 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:07.417068 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:05:13.237921 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:13.237883 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65"] Jul 14 19:05:14.085530 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:14.085495 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf"] Jul 14 19:05:19.487336 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.487300 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt"] Jul 14 19:05:19.490650 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.490632 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.493724 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.493694 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-52ef23\"/\"sub-test-model-52ef23-kserve-self-signed-certs\"" Jul 14 19:05:19.501928 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.501903 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt"] Jul 14 19:05:19.537605 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.537574 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb6mh\" (UniqueName: \"kubernetes.io/projected/bf2ed54d-87e6-4437-96a5-cef235bf5e80-kube-api-access-wb6mh\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.537779 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.537616 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-model-cache\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.537779 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.537725 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-dshm\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.537779 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.537755 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tmp-dir\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.537912 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.537786 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-home\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.537912 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.537809 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tls-certs\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.638806 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.638768 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-model-cache\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.638991 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.638828 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-dshm\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.638991 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.638852 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tmp-dir\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.638991 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.638882 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-home\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.638991 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.638901 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tls-certs\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.638991 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.638966 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb6mh\" (UniqueName: \"kubernetes.io/projected/bf2ed54d-87e6-4437-96a5-cef235bf5e80-kube-api-access-wb6mh\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.639243 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.639212 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-model-cache\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.639345 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.639320 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-home\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.639422 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.639406 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tmp-dir\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.641018 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.641000 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-dshm\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.641445 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.641421 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tls-certs\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.647927 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.647902 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb6mh\" (UniqueName: \"kubernetes.io/projected/bf2ed54d-87e6-4437-96a5-cef235bf5e80-kube-api-access-wb6mh\") pod \"sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.800094 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.800058 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:19.937351 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:19.937324 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt"] Jul 14 19:05:19.939900 ip-10-0-130-79 kubenswrapper[2569]: W0714 19:05:19.939869 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbf2ed54d_87e6_4437_96a5_cef235bf5e80.slice/crio-74e9f85147ab79c1e9a24de11c03f8826a5031374cdce47919acd8a596667537 WatchSource:0}: Error finding container 74e9f85147ab79c1e9a24de11c03f8826a5031374cdce47919acd8a596667537: Status 404 returned error can't find the container with id 74e9f85147ab79c1e9a24de11c03f8826a5031374cdce47919acd8a596667537 Jul 14 19:05:20.754076 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:20.754034 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" event={"ID":"bf2ed54d-87e6-4437-96a5-cef235bf5e80","Type":"ContainerStarted","Data":"04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f"} Jul 14 19:05:20.754568 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:20.754085 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" event={"ID":"bf2ed54d-87e6-4437-96a5-cef235bf5e80","Type":"ContainerStarted","Data":"74e9f85147ab79c1e9a24de11c03f8826a5031374cdce47919acd8a596667537"} Jul 14 19:05:20.778842 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:20.778783 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" podStartSLOduration=1.7787661030000002 podStartE2EDuration="1.778766103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:05:20.776543572 +0000 UTC m=+2614.834211622" watchObservedRunningTime="2026-07-14 19:05:20.778766103 +0000 UTC m=+2614.836434151" Jul 14 19:05:25.187044 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:25.187009 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:05:28.257661 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.257619 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" podUID="610d0b3a-497d-4a9a-a732-cd306876e8b9" containerName="main" containerID="cri-o://1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46" gracePeriod=45 Jul 14 19:05:28.505943 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.505917 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:05:28.620649 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.620533 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610d0b3a-497d-4a9a-a732-cd306876e8b9-tls-certs\") pod \"610d0b3a-497d-4a9a-a732-cd306876e8b9\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " Jul 14 19:05:28.620649 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.620585 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-tmp-dir\") pod \"610d0b3a-497d-4a9a-a732-cd306876e8b9\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " Jul 14 19:05:28.620649 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.620613 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-model-cache\") pod \"610d0b3a-497d-4a9a-a732-cd306876e8b9\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " Jul 14 19:05:28.620649 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.620629 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k975c\" (UniqueName: \"kubernetes.io/projected/610d0b3a-497d-4a9a-a732-cd306876e8b9-kube-api-access-k975c\") pod \"610d0b3a-497d-4a9a-a732-cd306876e8b9\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " Jul 14 19:05:28.620649 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.620668 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-dshm\") pod \"610d0b3a-497d-4a9a-a732-cd306876e8b9\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " Jul 14 19:05:28.621039 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.620692 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-home\") pod \"610d0b3a-497d-4a9a-a732-cd306876e8b9\" (UID: \"610d0b3a-497d-4a9a-a732-cd306876e8b9\") " Jul 14 19:05:28.621039 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.620805 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-model-cache" (OuterVolumeSpecName: "model-cache") pod "610d0b3a-497d-4a9a-a732-cd306876e8b9" (UID: "610d0b3a-497d-4a9a-a732-cd306876e8b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:05:28.621039 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.620841 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "610d0b3a-497d-4a9a-a732-cd306876e8b9" (UID: "610d0b3a-497d-4a9a-a732-cd306876e8b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:05:28.621039 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.621022 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-home" (OuterVolumeSpecName: "home") pod "610d0b3a-497d-4a9a-a732-cd306876e8b9" (UID: "610d0b3a-497d-4a9a-a732-cd306876e8b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:05:28.621218 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.621066 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:28.621218 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.621081 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:28.622869 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.622829 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/610d0b3a-497d-4a9a-a732-cd306876e8b9-kube-api-access-k975c" (OuterVolumeSpecName: "kube-api-access-k975c") pod "610d0b3a-497d-4a9a-a732-cd306876e8b9" (UID: "610d0b3a-497d-4a9a-a732-cd306876e8b9"). InnerVolumeSpecName "kube-api-access-k975c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:05:28.622869 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.622845 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/610d0b3a-497d-4a9a-a732-cd306876e8b9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "610d0b3a-497d-4a9a-a732-cd306876e8b9" (UID: "610d0b3a-497d-4a9a-a732-cd306876e8b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:05:28.623011 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.622858 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-dshm" (OuterVolumeSpecName: "dshm") pod "610d0b3a-497d-4a9a-a732-cd306876e8b9" (UID: "610d0b3a-497d-4a9a-a732-cd306876e8b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:05:28.722310 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.722245 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:28.722310 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.722304 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/610d0b3a-497d-4a9a-a732-cd306876e8b9-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:28.722310 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.722318 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k975c\" (UniqueName: \"kubernetes.io/projected/610d0b3a-497d-4a9a-a732-cd306876e8b9-kube-api-access-k975c\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:28.722557 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.722331 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/610d0b3a-497d-4a9a-a732-cd306876e8b9-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:28.797546 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.797514 2569 generic.go:356] "Generic (PLEG): container finished" podID="610d0b3a-497d-4a9a-a732-cd306876e8b9" containerID="1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46" exitCode=0 Jul 14 19:05:28.797736 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.797566 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" Jul 14 19:05:28.797736 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.797600 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" event={"ID":"610d0b3a-497d-4a9a-a732-cd306876e8b9","Type":"ContainerDied","Data":"1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46"} Jul 14 19:05:28.797736 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.797643 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65" event={"ID":"610d0b3a-497d-4a9a-a732-cd306876e8b9","Type":"ContainerDied","Data":"579c4256bc5d02e63aa1a90eb89303ee855fdcb25e870745b7f8f0a506ed20ab"} Jul 14 19:05:28.797736 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.797662 2569 scope.go:117] "RemoveContainer" containerID="1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46" Jul 14 19:05:28.809828 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.809803 2569 scope.go:117] "RemoveContainer" containerID="1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46" Jul 14 19:05:28.810130 ip-10-0-130-79 kubenswrapper[2569]: E0714 19:05:28.810107 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46\": container with ID starting with 1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46 not found: ID does not exist" containerID="1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46" Jul 14 19:05:28.810188 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.810139 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46"} err="failed to get container status \"1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46\": rpc error: code = NotFound desc = could not find container \"1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46\": container with ID starting with 1d1f71e7165a7a0ec780efe51c256e96e434f940ccad402120a8302896fa0e46 not found: ID does not exist" Jul 14 19:05:28.821135 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.821079 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65"] Jul 14 19:05:28.824041 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:28.824013 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/auth-test-model-52ef23-kserve-56cc7d88d7-zth65"] Jul 14 19:05:29.105599 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.105559 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" podUID="f750c968-7622-41b1-bf0c-a51b49aad79e" containerName="main" containerID="cri-o://a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd" gracePeriod=45 Jul 14 19:05:29.357553 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.357488 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:05:29.428206 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428164 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-model-cache\") pod \"f750c968-7622-41b1-bf0c-a51b49aad79e\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " Jul 14 19:05:29.428436 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428216 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-home\") pod \"f750c968-7622-41b1-bf0c-a51b49aad79e\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " Jul 14 19:05:29.428436 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428274 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2xdqq\" (UniqueName: \"kubernetes.io/projected/f750c968-7622-41b1-bf0c-a51b49aad79e-kube-api-access-2xdqq\") pod \"f750c968-7622-41b1-bf0c-a51b49aad79e\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " Jul 14 19:05:29.428436 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428322 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f750c968-7622-41b1-bf0c-a51b49aad79e-tls-certs\") pod \"f750c968-7622-41b1-bf0c-a51b49aad79e\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " Jul 14 19:05:29.428436 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428354 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-tmp-dir\") pod \"f750c968-7622-41b1-bf0c-a51b49aad79e\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " Jul 14 19:05:29.428651 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428453 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-dshm\") pod \"f750c968-7622-41b1-bf0c-a51b49aad79e\" (UID: \"f750c968-7622-41b1-bf0c-a51b49aad79e\") " Jul 14 19:05:29.428651 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428478 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-model-cache" (OuterVolumeSpecName: "model-cache") pod "f750c968-7622-41b1-bf0c-a51b49aad79e" (UID: "f750c968-7622-41b1-bf0c-a51b49aad79e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:05:29.428651 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428521 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-home" (OuterVolumeSpecName: "home") pod "f750c968-7622-41b1-bf0c-a51b49aad79e" (UID: "f750c968-7622-41b1-bf0c-a51b49aad79e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:05:29.428795 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428700 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:29.428795 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428709 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f750c968-7622-41b1-bf0c-a51b49aad79e" (UID: "f750c968-7622-41b1-bf0c-a51b49aad79e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:05:29.428795 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.428716 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:29.430645 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.430620 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-dshm" (OuterVolumeSpecName: "dshm") pod "f750c968-7622-41b1-bf0c-a51b49aad79e" (UID: "f750c968-7622-41b1-bf0c-a51b49aad79e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:05:29.430749 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.430694 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f750c968-7622-41b1-bf0c-a51b49aad79e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f750c968-7622-41b1-bf0c-a51b49aad79e" (UID: "f750c968-7622-41b1-bf0c-a51b49aad79e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:05:29.430791 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.430746 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f750c968-7622-41b1-bf0c-a51b49aad79e-kube-api-access-2xdqq" (OuterVolumeSpecName: "kube-api-access-2xdqq") pod "f750c968-7622-41b1-bf0c-a51b49aad79e" (UID: "f750c968-7622-41b1-bf0c-a51b49aad79e"). InnerVolumeSpecName "kube-api-access-2xdqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:05:29.529853 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.529814 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f750c968-7622-41b1-bf0c-a51b49aad79e-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:29.529853 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.529846 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:29.529853 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.529854 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f750c968-7622-41b1-bf0c-a51b49aad79e-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:29.529853 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.529862 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2xdqq\" (UniqueName: \"kubernetes.io/projected/f750c968-7622-41b1-bf0c-a51b49aad79e-kube-api-access-2xdqq\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:05:29.800500 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.800461 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:29.800500 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.800511 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:29.805981 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.805949 2569 generic.go:356] "Generic (PLEG): container finished" podID="f750c968-7622-41b1-bf0c-a51b49aad79e" containerID="a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd" exitCode=0 Jul 14 19:05:29.806119 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.806004 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" Jul 14 19:05:29.806119 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.806002 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" event={"ID":"f750c968-7622-41b1-bf0c-a51b49aad79e","Type":"ContainerDied","Data":"a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd"} Jul 14 19:05:29.806231 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.806125 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf" event={"ID":"f750c968-7622-41b1-bf0c-a51b49aad79e","Type":"ContainerDied","Data":"43ad50de5f8e514c742ef41af10314dd8d5ce49195549fe0a38b4e947ab4f738"} Jul 14 19:05:29.806231 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.806149 2569 scope.go:117] "RemoveContainer" containerID="a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd" Jul 14 19:05:29.814819 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.814796 2569 scope.go:117] "RemoveContainer" containerID="a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd" Jul 14 19:05:29.815107 ip-10-0-130-79 kubenswrapper[2569]: E0714 19:05:29.815089 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd\": container with ID starting with a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd not found: ID does not exist" containerID="a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd" Jul 14 19:05:29.815158 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.815113 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd"} err="failed to get container status \"a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd\": rpc error: code = NotFound desc = could not find container \"a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd\": container with ID starting with a316a068ea57fed5c0cb43a16fbcdb4a75633332dc899efb9e2485c4744c59fd not found: ID does not exist" Jul 14 19:05:29.817136 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.817116 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:05:29.832529 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.832500 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf"] Jul 14 19:05:29.837684 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:29.837657 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/auth-test-model-10fa01-kserve-78b676ddf5-6lbdf"] Jul 14 19:05:30.482587 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:30.482551 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="610d0b3a-497d-4a9a-a732-cd306876e8b9" path="/var/lib/kubelet/pods/610d0b3a-497d-4a9a-a732-cd306876e8b9/volumes" Jul 14 19:05:30.482980 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:30.482952 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f750c968-7622-41b1-bf0c-a51b49aad79e" path="/var/lib/kubelet/pods/f750c968-7622-41b1-bf0c-a51b49aad79e/volumes" Jul 14 19:05:30.823593 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:05:30.823562 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:06:07.187052 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.187015 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b"] Jul 14 19:06:07.187526 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.187358 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f750c968-7622-41b1-bf0c-a51b49aad79e" containerName="main" Jul 14 19:06:07.187526 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.187371 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="f750c968-7622-41b1-bf0c-a51b49aad79e" containerName="main" Jul 14 19:06:07.187526 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.187391 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="610d0b3a-497d-4a9a-a732-cd306876e8b9" containerName="main" Jul 14 19:06:07.187526 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.187396 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="610d0b3a-497d-4a9a-a732-cd306876e8b9" containerName="main" Jul 14 19:06:07.187526 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.187454 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="f750c968-7622-41b1-bf0c-a51b49aad79e" containerName="main" Jul 14 19:06:07.187526 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.187463 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="610d0b3a-497d-4a9a-a732-cd306876e8b9" containerName="main" Jul 14 19:06:07.190355 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.190337 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.193465 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.193443 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"kube-root-ca.crt\"" Jul 14 19:06:07.194901 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.194880 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"openshift-service-ca.crt\"" Jul 14 19:06:07.194999 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.194914 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"sub-test-model-10fa01-kserve-self-signed-certs\"" Jul 14 19:06:07.195362 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.195346 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"default-dockercfg-4hckz\"" Jul 14 19:06:07.202572 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.202552 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b"] Jul 14 19:06:07.244779 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.244748 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-dshm\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.244944 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.244819 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9vmr\" (UniqueName: \"kubernetes.io/projected/05793469-6cec-42cf-8f59-44fdcd5f56fa-kube-api-access-z9vmr\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.244944 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.244850 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05793469-6cec-42cf-8f59-44fdcd5f56fa-tls-certs\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.244944 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.244870 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-tmp-dir\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.244944 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.244892 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-home\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.245090 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.244941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-model-cache\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345370 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345336 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9vmr\" (UniqueName: \"kubernetes.io/projected/05793469-6cec-42cf-8f59-44fdcd5f56fa-kube-api-access-z9vmr\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345370 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345376 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05793469-6cec-42cf-8f59-44fdcd5f56fa-tls-certs\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345634 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345397 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-tmp-dir\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345634 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345420 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-home\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345634 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345455 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-model-cache\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345634 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345494 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-dshm\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345865 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345839 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-tmp-dir\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345924 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345846 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-model-cache\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.345924 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.345888 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-home\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.347761 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.347744 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-dshm\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.348022 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.347999 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05793469-6cec-42cf-8f59-44fdcd5f56fa-tls-certs\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.355545 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.355525 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9vmr\" (UniqueName: \"kubernetes.io/projected/05793469-6cec-42cf-8f59-44fdcd5f56fa-kube-api-access-z9vmr\") pod \"sub-test-model-10fa01-kserve-6fd6857c-zqf9b\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.500114 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.500029 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:07.640139 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.640112 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b"] Jul 14 19:06:07.641613 ip-10-0-130-79 kubenswrapper[2569]: W0714 19:06:07.641583 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05793469_6cec_42cf_8f59_44fdcd5f56fa.slice/crio-113e4bbd321fe314cf4f53b20d5b2491e5720ef558f946bf364875e36ad6e790 WatchSource:0}: Error finding container 113e4bbd321fe314cf4f53b20d5b2491e5720ef558f946bf364875e36ad6e790: Status 404 returned error can't find the container with id 113e4bbd321fe314cf4f53b20d5b2491e5720ef558f946bf364875e36ad6e790 Jul 14 19:06:07.992435 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.992399 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" event={"ID":"05793469-6cec-42cf-8f59-44fdcd5f56fa","Type":"ContainerStarted","Data":"f2aee9f919e62ebc2c34f3d9a3d38d4bb6984eaebc4231c8e16aa69cc1a62aae"} Jul 14 19:06:07.992607 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:07.992444 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" event={"ID":"05793469-6cec-42cf-8f59-44fdcd5f56fa","Type":"ContainerStarted","Data":"113e4bbd321fe314cf4f53b20d5b2491e5720ef558f946bf364875e36ad6e790"} Jul 14 19:06:08.012577 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:08.012494 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" podStartSLOduration=1.012477458 podStartE2EDuration="1.012477458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:06:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:06:08.01134697 +0000 UTC m=+2662.069015019" watchObservedRunningTime="2026-07-14 19:06:08.012477458 +0000 UTC m=+2662.070145506" Jul 14 19:06:11.398135 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:11.398096 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:06:17.500815 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:17.500776 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:17.501242 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:17.500837 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:17.514287 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:17.514243 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:18.059180 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:18.059153 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:06:31.995271 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:31.995213 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:06:48.269986 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:48.269905 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt"] Jul 14 19:06:49.086713 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:49.086673 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b"] Jul 14 19:06:52.798954 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:52.798916 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:06:56.957841 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:56.957807 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd"] Jul 14 19:06:56.961039 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:56.961022 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:56.963822 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:56.963803 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-52ef23\"/\"rate-test-model-52ef23-kserve-self-signed-certs\"" Jul 14 19:06:56.971841 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:56.971812 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd"] Jul 14 19:06:57.071123 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.071088 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-tmp-dir\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.071355 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.071134 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-home\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.071355 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.071156 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-dshm\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.071355 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.071199 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b231ee2f-006b-410f-9437-9f9281b5ea56-tls-certs\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.071355 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.071335 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgjfp\" (UniqueName: \"kubernetes.io/projected/b231ee2f-006b-410f-9437-9f9281b5ea56-kube-api-access-sgjfp\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.071560 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.071395 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-model-cache\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.172451 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172414 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-tmp-dir\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.172661 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172466 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-home\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.172661 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172491 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-dshm\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.172661 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172508 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b231ee2f-006b-410f-9437-9f9281b5ea56-tls-certs\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.172661 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172533 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgjfp\" (UniqueName: \"kubernetes.io/projected/b231ee2f-006b-410f-9437-9f9281b5ea56-kube-api-access-sgjfp\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.172661 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172579 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-model-cache\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.172956 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172819 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-tmp-dir\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.172956 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172903 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-home\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.173068 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.172967 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-model-cache\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.174853 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.174827 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-dshm\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.175188 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.175172 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b231ee2f-006b-410f-9437-9f9281b5ea56-tls-certs\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.182099 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.182079 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgjfp\" (UniqueName: \"kubernetes.io/projected/b231ee2f-006b-410f-9437-9f9281b5ea56-kube-api-access-sgjfp\") pod \"rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.270543 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.270513 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:06:57.407419 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:57.407287 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd"] Jul 14 19:06:57.410069 ip-10-0-130-79 kubenswrapper[2569]: W0714 19:06:57.410043 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb231ee2f_006b_410f_9437_9f9281b5ea56.slice/crio-f8641d3c808defed9e502db9212679aa30d679c025bcc230136069f87ee62ae5 WatchSource:0}: Error finding container f8641d3c808defed9e502db9212679aa30d679c025bcc230136069f87ee62ae5: Status 404 returned error can't find the container with id f8641d3c808defed9e502db9212679aa30d679c025bcc230136069f87ee62ae5 Jul 14 19:06:58.243086 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:58.243047 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" event={"ID":"b231ee2f-006b-410f-9437-9f9281b5ea56","Type":"ContainerStarted","Data":"a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31"} Jul 14 19:06:58.243540 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:58.243100 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" event={"ID":"b231ee2f-006b-410f-9437-9f9281b5ea56","Type":"ContainerStarted","Data":"f8641d3c808defed9e502db9212679aa30d679c025bcc230136069f87ee62ae5"} Jul 14 19:06:58.264094 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:06:58.264044 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" podStartSLOduration=2.264028217 podStartE2EDuration="2.264028217s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:06:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:06:58.263102162 +0000 UTC m=+2712.320770215" watchObservedRunningTime="2026-07-14 19:06:58.264028217 +0000 UTC m=+2712.321696306" Jul 14 19:07:03.289615 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.289570 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" podUID="bf2ed54d-87e6-4437-96a5-cef235bf5e80" containerName="main" containerID="cri-o://04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f" gracePeriod=45 Jul 14 19:07:03.552243 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.552180 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:07:03.736577 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.736543 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tmp-dir\") pod \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " Jul 14 19:07:03.736781 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.736605 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tls-certs\") pod \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " Jul 14 19:07:03.736781 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.736644 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-model-cache\") pod \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " Jul 14 19:07:03.736781 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.736689 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wb6mh\" (UniqueName: \"kubernetes.io/projected/bf2ed54d-87e6-4437-96a5-cef235bf5e80-kube-api-access-wb6mh\") pod \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " Jul 14 19:07:03.736781 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.736726 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-home\") pod \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " Jul 14 19:07:03.736781 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.736760 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-dshm\") pod \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\" (UID: \"bf2ed54d-87e6-4437-96a5-cef235bf5e80\") " Jul 14 19:07:03.737768 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.736807 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bf2ed54d-87e6-4437-96a5-cef235bf5e80" (UID: "bf2ed54d-87e6-4437-96a5-cef235bf5e80"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:07:03.737768 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.736853 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-model-cache" (OuterVolumeSpecName: "model-cache") pod "bf2ed54d-87e6-4437-96a5-cef235bf5e80" (UID: "bf2ed54d-87e6-4437-96a5-cef235bf5e80"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:07:03.737768 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.737080 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:03.737768 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.737100 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:03.737768 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.737334 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-home" (OuterVolumeSpecName: "home") pod "bf2ed54d-87e6-4437-96a5-cef235bf5e80" (UID: "bf2ed54d-87e6-4437-96a5-cef235bf5e80"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:07:03.739447 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.739420 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bf2ed54d-87e6-4437-96a5-cef235bf5e80" (UID: "bf2ed54d-87e6-4437-96a5-cef235bf5e80"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:07:03.739573 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.739560 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-dshm" (OuterVolumeSpecName: "dshm") pod "bf2ed54d-87e6-4437-96a5-cef235bf5e80" (UID: "bf2ed54d-87e6-4437-96a5-cef235bf5e80"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:07:03.739687 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.739663 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf2ed54d-87e6-4437-96a5-cef235bf5e80-kube-api-access-wb6mh" (OuterVolumeSpecName: "kube-api-access-wb6mh") pod "bf2ed54d-87e6-4437-96a5-cef235bf5e80" (UID: "bf2ed54d-87e6-4437-96a5-cef235bf5e80"). InnerVolumeSpecName "kube-api-access-wb6mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:07:03.838572 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.838484 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb6mh\" (UniqueName: \"kubernetes.io/projected/bf2ed54d-87e6-4437-96a5-cef235bf5e80-kube-api-access-wb6mh\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:03.838572 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.838523 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:03.838572 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.838536 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bf2ed54d-87e6-4437-96a5-cef235bf5e80-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:03.838572 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:03.838548 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bf2ed54d-87e6-4437-96a5-cef235bf5e80-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:04.106688 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.106571 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" podUID="05793469-6cec-42cf-8f59-44fdcd5f56fa" containerName="main" containerID="cri-o://f2aee9f919e62ebc2c34f3d9a3d38d4bb6984eaebc4231c8e16aa69cc1a62aae" gracePeriod=45 Jul 14 19:07:04.274569 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.274535 2569 generic.go:356] "Generic (PLEG): container finished" podID="bf2ed54d-87e6-4437-96a5-cef235bf5e80" containerID="04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f" exitCode=0 Jul 14 19:07:04.274733 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.274599 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" Jul 14 19:07:04.274794 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.274757 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" event={"ID":"bf2ed54d-87e6-4437-96a5-cef235bf5e80","Type":"ContainerDied","Data":"04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f"} Jul 14 19:07:04.274860 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.274799 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt" event={"ID":"bf2ed54d-87e6-4437-96a5-cef235bf5e80","Type":"ContainerDied","Data":"74e9f85147ab79c1e9a24de11c03f8826a5031374cdce47919acd8a596667537"} Jul 14 19:07:04.274860 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.274820 2569 scope.go:117] "RemoveContainer" containerID="04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f" Jul 14 19:07:04.276926 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.276881 2569 generic.go:356] "Generic (PLEG): container finished" podID="05793469-6cec-42cf-8f59-44fdcd5f56fa" containerID="f2aee9f919e62ebc2c34f3d9a3d38d4bb6984eaebc4231c8e16aa69cc1a62aae" exitCode=0 Jul 14 19:07:04.277045 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.276917 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" event={"ID":"05793469-6cec-42cf-8f59-44fdcd5f56fa","Type":"ContainerDied","Data":"f2aee9f919e62ebc2c34f3d9a3d38d4bb6984eaebc4231c8e16aa69cc1a62aae"} Jul 14 19:07:04.284520 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.284500 2569 scope.go:117] "RemoveContainer" containerID="04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f" Jul 14 19:07:04.284807 ip-10-0-130-79 kubenswrapper[2569]: E0714 19:07:04.284789 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f\": container with ID starting with 04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f not found: ID does not exist" containerID="04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f" Jul 14 19:07:04.284895 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.284819 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f"} err="failed to get container status \"04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f\": rpc error: code = NotFound desc = could not find container \"04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f\": container with ID starting with 04eb727ae0a74e4e03cc3f9c8dadfbee4a02938e02904abd05261186b45cf09f not found: ID does not exist" Jul 14 19:07:04.300326 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.300295 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt"] Jul 14 19:07:04.307222 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.307195 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/sub-test-model-52ef23-kserve-6577b4d4cc-4xrqt"] Jul 14 19:07:04.361780 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.361722 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:07:04.482138 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.482106 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf2ed54d-87e6-4437-96a5-cef235bf5e80" path="/var/lib/kubelet/pods/bf2ed54d-87e6-4437-96a5-cef235bf5e80/volumes" Jul 14 19:07:04.544172 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544141 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-model-cache\") pod \"05793469-6cec-42cf-8f59-44fdcd5f56fa\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " Jul 14 19:07:04.544371 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544216 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z9vmr\" (UniqueName: \"kubernetes.io/projected/05793469-6cec-42cf-8f59-44fdcd5f56fa-kube-api-access-z9vmr\") pod \"05793469-6cec-42cf-8f59-44fdcd5f56fa\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " Jul 14 19:07:04.544371 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544247 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-dshm\") pod \"05793469-6cec-42cf-8f59-44fdcd5f56fa\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " Jul 14 19:07:04.544371 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544326 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-home\") pod \"05793469-6cec-42cf-8f59-44fdcd5f56fa\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " Jul 14 19:07:04.544568 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544388 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-tmp-dir\") pod \"05793469-6cec-42cf-8f59-44fdcd5f56fa\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " Jul 14 19:07:04.544568 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544449 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05793469-6cec-42cf-8f59-44fdcd5f56fa-tls-certs\") pod \"05793469-6cec-42cf-8f59-44fdcd5f56fa\" (UID: \"05793469-6cec-42cf-8f59-44fdcd5f56fa\") " Jul 14 19:07:04.544568 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544462 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-model-cache" (OuterVolumeSpecName: "model-cache") pod "05793469-6cec-42cf-8f59-44fdcd5f56fa" (UID: "05793469-6cec-42cf-8f59-44fdcd5f56fa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:07:04.544722 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544664 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-home" (OuterVolumeSpecName: "home") pod "05793469-6cec-42cf-8f59-44fdcd5f56fa" (UID: "05793469-6cec-42cf-8f59-44fdcd5f56fa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:07:04.544722 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544671 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "05793469-6cec-42cf-8f59-44fdcd5f56fa" (UID: "05793469-6cec-42cf-8f59-44fdcd5f56fa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:07:04.544722 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.544715 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:04.546597 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.546568 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05793469-6cec-42cf-8f59-44fdcd5f56fa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "05793469-6cec-42cf-8f59-44fdcd5f56fa" (UID: "05793469-6cec-42cf-8f59-44fdcd5f56fa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:07:04.546846 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.546827 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-dshm" (OuterVolumeSpecName: "dshm") pod "05793469-6cec-42cf-8f59-44fdcd5f56fa" (UID: "05793469-6cec-42cf-8f59-44fdcd5f56fa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:07:04.546920 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.546841 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05793469-6cec-42cf-8f59-44fdcd5f56fa-kube-api-access-z9vmr" (OuterVolumeSpecName: "kube-api-access-z9vmr") pod "05793469-6cec-42cf-8f59-44fdcd5f56fa" (UID: "05793469-6cec-42cf-8f59-44fdcd5f56fa"). InnerVolumeSpecName "kube-api-access-z9vmr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:07:04.645536 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.645456 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9vmr\" (UniqueName: \"kubernetes.io/projected/05793469-6cec-42cf-8f59-44fdcd5f56fa-kube-api-access-z9vmr\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:04.645536 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.645481 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:04.645536 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.645490 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:04.645536 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.645498 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05793469-6cec-42cf-8f59-44fdcd5f56fa-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:04.645536 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:04.645506 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05793469-6cec-42cf-8f59-44fdcd5f56fa-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:07:05.288759 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:05.288728 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" Jul 14 19:07:05.288957 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:05.288729 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b" event={"ID":"05793469-6cec-42cf-8f59-44fdcd5f56fa","Type":"ContainerDied","Data":"113e4bbd321fe314cf4f53b20d5b2491e5720ef558f946bf364875e36ad6e790"} Jul 14 19:07:05.288957 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:05.288860 2569 scope.go:117] "RemoveContainer" containerID="f2aee9f919e62ebc2c34f3d9a3d38d4bb6984eaebc4231c8e16aa69cc1a62aae" Jul 14 19:07:05.311993 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:05.311964 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b"] Jul 14 19:07:05.314105 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:05.314084 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/sub-test-model-10fa01-kserve-6fd6857c-zqf9b"] Jul 14 19:07:06.481710 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:06.481678 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05793469-6cec-42cf-8f59-44fdcd5f56fa" path="/var/lib/kubelet/pods/05793469-6cec-42cf-8f59-44fdcd5f56fa/volumes" Jul 14 19:07:07.270753 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:07.270709 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:07:07.270950 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:07.270766 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:07:07.283085 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:07.283059 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:07:07.312770 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:07.312740 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:07:09.696977 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:09.696939 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:07:25.284080 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.284046 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d"] Jul 14 19:07:25.284515 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.284403 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf2ed54d-87e6-4437-96a5-cef235bf5e80" containerName="main" Jul 14 19:07:25.284515 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.284415 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf2ed54d-87e6-4437-96a5-cef235bf5e80" containerName="main" Jul 14 19:07:25.284515 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.284426 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05793469-6cec-42cf-8f59-44fdcd5f56fa" containerName="main" Jul 14 19:07:25.284515 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.284431 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="05793469-6cec-42cf-8f59-44fdcd5f56fa" containerName="main" Jul 14 19:07:25.284515 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.284488 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="05793469-6cec-42cf-8f59-44fdcd5f56fa" containerName="main" Jul 14 19:07:25.284515 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.284496 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf2ed54d-87e6-4437-96a5-cef235bf5e80" containerName="main" Jul 14 19:07:25.287213 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.287197 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.290036 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.290008 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"openshift-service-ca.crt\"" Jul 14 19:07:25.290173 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.290042 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"kube-root-ca.crt\"" Jul 14 19:07:25.290173 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.290062 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"default-dockercfg-4hckz\"" Jul 14 19:07:25.291432 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.291414 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-10fa01\"/\"rate-test-model-10fa01-kserve-self-signed-certs\"" Jul 14 19:07:25.300362 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.300339 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d"] Jul 14 19:07:25.422794 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.422760 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-dshm\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.422977 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.422802 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-model-cache\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.422977 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.422841 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cwgf\" (UniqueName: \"kubernetes.io/projected/3deb05fb-09ec-48be-8267-d13a85c6a1eb-kube-api-access-9cwgf\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.422977 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.422869 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tls-certs\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.422977 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.422906 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-home\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.422977 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.422941 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tmp-dir\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.523963 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.523922 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-dshm\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.524135 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.523975 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-model-cache\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.524135 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.524096 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cwgf\" (UniqueName: \"kubernetes.io/projected/3deb05fb-09ec-48be-8267-d13a85c6a1eb-kube-api-access-9cwgf\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.524236 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.524148 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tls-certs\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.524236 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.524197 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-home\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.524236 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.524223 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tmp-dir\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.524442 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.524369 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-model-cache\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.524567 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.524545 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-home\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.524622 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.524562 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tmp-dir\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.526457 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.526434 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-dshm\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.526689 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.526669 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tls-certs\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.533977 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.533951 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cwgf\" (UniqueName: \"kubernetes.io/projected/3deb05fb-09ec-48be-8267-d13a85c6a1eb-kube-api-access-9cwgf\") pod \"rate-test-model-10fa01-kserve-755c795c87-c8j7d\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.596000 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.595911 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:25.726818 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:25.726796 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d"] Jul 14 19:07:25.728693 ip-10-0-130-79 kubenswrapper[2569]: W0714 19:07:25.728658 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3deb05fb_09ec_48be_8267_d13a85c6a1eb.slice/crio-33dd03de3b783ac12a28b918f2d6f8e302c0d4e7858c4b491d4efa3a1884fb79 WatchSource:0}: Error finding container 33dd03de3b783ac12a28b918f2d6f8e302c0d4e7858c4b491d4efa3a1884fb79: Status 404 returned error can't find the container with id 33dd03de3b783ac12a28b918f2d6f8e302c0d4e7858c4b491d4efa3a1884fb79 Jul 14 19:07:26.394702 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:26.394665 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" event={"ID":"3deb05fb-09ec-48be-8267-d13a85c6a1eb","Type":"ContainerStarted","Data":"fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a"} Jul 14 19:07:26.395102 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:26.394719 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" event={"ID":"3deb05fb-09ec-48be-8267-d13a85c6a1eb","Type":"ContainerStarted","Data":"33dd03de3b783ac12a28b918f2d6f8e302c0d4e7858c4b491d4efa3a1884fb79"} Jul 14 19:07:26.419872 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:26.419825 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" podStartSLOduration=1.419806603 podStartE2EDuration="1.419806603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 19:07:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:07:26.416898666 +0000 UTC m=+2740.474566764" watchObservedRunningTime="2026-07-14 19:07:26.419806603 +0000 UTC m=+2740.477474651" Jul 14 19:07:29.299380 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:29.299343 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:07:35.597029 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:35.596979 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:35.597559 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:35.597057 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:35.610120 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:35.610096 2569 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:36.456270 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:36.456211 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:07:43.914861 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:07:43.914818 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:08:00.594924 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:00.594886 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:08:08.109525 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:08.109445 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd"] Jul 14 19:08:08.992023 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:08.991990 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d"] Jul 14 19:08:15.489666 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:15.489627 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:08:23.128840 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.128793 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" podUID="b231ee2f-006b-410f-9437-9f9281b5ea56" containerName="main" containerID="cri-o://a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31" gracePeriod=45 Jul 14 19:08:23.380246 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.380182 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:08:23.545982 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.545944 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-home\") pod \"b231ee2f-006b-410f-9437-9f9281b5ea56\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " Jul 14 19:08:23.545982 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.545984 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-dshm\") pod \"b231ee2f-006b-410f-9437-9f9281b5ea56\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " Jul 14 19:08:23.546239 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546013 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b231ee2f-006b-410f-9437-9f9281b5ea56-tls-certs\") pod \"b231ee2f-006b-410f-9437-9f9281b5ea56\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " Jul 14 19:08:23.546239 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546070 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-tmp-dir\") pod \"b231ee2f-006b-410f-9437-9f9281b5ea56\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " Jul 14 19:08:23.546239 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546095 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-model-cache\") pod \"b231ee2f-006b-410f-9437-9f9281b5ea56\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " Jul 14 19:08:23.546445 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546277 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgjfp\" (UniqueName: \"kubernetes.io/projected/b231ee2f-006b-410f-9437-9f9281b5ea56-kube-api-access-sgjfp\") pod \"b231ee2f-006b-410f-9437-9f9281b5ea56\" (UID: \"b231ee2f-006b-410f-9437-9f9281b5ea56\") " Jul 14 19:08:23.546445 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546304 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-home" (OuterVolumeSpecName: "home") pod "b231ee2f-006b-410f-9437-9f9281b5ea56" (UID: "b231ee2f-006b-410f-9437-9f9281b5ea56"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:08:23.546445 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546380 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-model-cache" (OuterVolumeSpecName: "model-cache") pod "b231ee2f-006b-410f-9437-9f9281b5ea56" (UID: "b231ee2f-006b-410f-9437-9f9281b5ea56"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:08:23.546445 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546400 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b231ee2f-006b-410f-9437-9f9281b5ea56" (UID: "b231ee2f-006b-410f-9437-9f9281b5ea56"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:08:23.546634 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546616 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:23.546677 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546641 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:23.546677 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.546655 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:23.548108 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.548082 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-dshm" (OuterVolumeSpecName: "dshm") pod "b231ee2f-006b-410f-9437-9f9281b5ea56" (UID: "b231ee2f-006b-410f-9437-9f9281b5ea56"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:08:23.548301 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.548274 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b231ee2f-006b-410f-9437-9f9281b5ea56-kube-api-access-sgjfp" (OuterVolumeSpecName: "kube-api-access-sgjfp") pod "b231ee2f-006b-410f-9437-9f9281b5ea56" (UID: "b231ee2f-006b-410f-9437-9f9281b5ea56"). InnerVolumeSpecName "kube-api-access-sgjfp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:23.548372 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.548320 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b231ee2f-006b-410f-9437-9f9281b5ea56-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b231ee2f-006b-410f-9437-9f9281b5ea56" (UID: "b231ee2f-006b-410f-9437-9f9281b5ea56"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:08:23.648279 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.648157 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgjfp\" (UniqueName: \"kubernetes.io/projected/b231ee2f-006b-410f-9437-9f9281b5ea56-kube-api-access-sgjfp\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:23.648279 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.648188 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b231ee2f-006b-410f-9437-9f9281b5ea56-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:23.648279 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.648199 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b231ee2f-006b-410f-9437-9f9281b5ea56-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:23.685209 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.685173 2569 generic.go:356] "Generic (PLEG): container finished" podID="b231ee2f-006b-410f-9437-9f9281b5ea56" containerID="a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31" exitCode=0 Jul 14 19:08:23.685392 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.685214 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" Jul 14 19:08:23.685392 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.685245 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" event={"ID":"b231ee2f-006b-410f-9437-9f9281b5ea56","Type":"ContainerDied","Data":"a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31"} Jul 14 19:08:23.685392 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.685299 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd" event={"ID":"b231ee2f-006b-410f-9437-9f9281b5ea56","Type":"ContainerDied","Data":"f8641d3c808defed9e502db9212679aa30d679c025bcc230136069f87ee62ae5"} Jul 14 19:08:23.685392 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.685318 2569 scope.go:117] "RemoveContainer" containerID="a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31" Jul 14 19:08:23.694908 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.694887 2569 scope.go:117] "RemoveContainer" containerID="a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31" Jul 14 19:08:23.695161 ip-10-0-130-79 kubenswrapper[2569]: E0714 19:08:23.695140 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31\": container with ID starting with a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31 not found: ID does not exist" containerID="a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31" Jul 14 19:08:23.695208 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.695170 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31"} err="failed to get container status \"a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31\": rpc error: code = NotFound desc = could not find container \"a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31\": container with ID starting with a233fdef63a0fc98669eaa651938d32a8932e0fb3305c58cd09619238e2d3f31 not found: ID does not exist" Jul 14 19:08:23.708383 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.708352 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd"] Jul 14 19:08:23.714606 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:23.714580 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-52ef23/rate-test-model-52ef23-kserve-5d9784dcc9-fm5cd"] Jul 14 19:08:24.011287 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.011234 2569 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" podUID="3deb05fb-09ec-48be-8267-d13a85c6a1eb" containerName="main" containerID="cri-o://fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a" gracePeriod=45 Jul 14 19:08:24.262077 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.262015 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:08:24.355189 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355157 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-model-cache\") pod \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " Jul 14 19:08:24.355429 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355235 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-home\") pod \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " Jul 14 19:08:24.355429 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355289 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-dshm\") pod \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " Jul 14 19:08:24.355429 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355322 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cwgf\" (UniqueName: \"kubernetes.io/projected/3deb05fb-09ec-48be-8267-d13a85c6a1eb-kube-api-access-9cwgf\") pod \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " Jul 14 19:08:24.355429 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355344 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tmp-dir\") pod \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " Jul 14 19:08:24.355429 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355379 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tls-certs\") pod \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\" (UID: \"3deb05fb-09ec-48be-8267-d13a85c6a1eb\") " Jul 14 19:08:24.355660 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355509 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-model-cache" (OuterVolumeSpecName: "model-cache") pod "3deb05fb-09ec-48be-8267-d13a85c6a1eb" (UID: "3deb05fb-09ec-48be-8267-d13a85c6a1eb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:08:24.355660 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355520 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-home" (OuterVolumeSpecName: "home") pod "3deb05fb-09ec-48be-8267-d13a85c6a1eb" (UID: "3deb05fb-09ec-48be-8267-d13a85c6a1eb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:08:24.355660 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355612 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3deb05fb-09ec-48be-8267-d13a85c6a1eb" (UID: "3deb05fb-09ec-48be-8267-d13a85c6a1eb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:08:24.355791 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355662 2569 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-model-cache\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:24.355791 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.355678 2569 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-home\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:24.357495 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.357471 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-dshm" (OuterVolumeSpecName: "dshm") pod "3deb05fb-09ec-48be-8267-d13a85c6a1eb" (UID: "3deb05fb-09ec-48be-8267-d13a85c6a1eb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 14 19:08:24.357597 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.357557 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3deb05fb-09ec-48be-8267-d13a85c6a1eb-kube-api-access-9cwgf" (OuterVolumeSpecName: "kube-api-access-9cwgf") pod "3deb05fb-09ec-48be-8267-d13a85c6a1eb" (UID: "3deb05fb-09ec-48be-8267-d13a85c6a1eb"). InnerVolumeSpecName "kube-api-access-9cwgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:24.357597 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.357572 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3deb05fb-09ec-48be-8267-d13a85c6a1eb" (UID: "3deb05fb-09ec-48be-8267-d13a85c6a1eb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 14 19:08:24.456636 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.456594 2569 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-dshm\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:24.456636 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.456628 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cwgf\" (UniqueName: \"kubernetes.io/projected/3deb05fb-09ec-48be-8267-d13a85c6a1eb-kube-api-access-9cwgf\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:24.456636 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.456640 2569 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tmp-dir\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:24.456636 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.456648 2569 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3deb05fb-09ec-48be-8267-d13a85c6a1eb-tls-certs\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:24.482415 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.482388 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b231ee2f-006b-410f-9437-9f9281b5ea56" path="/var/lib/kubelet/pods/b231ee2f-006b-410f-9437-9f9281b5ea56/volumes" Jul 14 19:08:24.691947 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.691862 2569 generic.go:356] "Generic (PLEG): container finished" podID="3deb05fb-09ec-48be-8267-d13a85c6a1eb" containerID="fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a" exitCode=0 Jul 14 19:08:24.691947 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.691909 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" Jul 14 19:08:24.691947 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.691931 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" event={"ID":"3deb05fb-09ec-48be-8267-d13a85c6a1eb","Type":"ContainerDied","Data":"fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a"} Jul 14 19:08:24.692224 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.691968 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d" event={"ID":"3deb05fb-09ec-48be-8267-d13a85c6a1eb","Type":"ContainerDied","Data":"33dd03de3b783ac12a28b918f2d6f8e302c0d4e7858c4b491d4efa3a1884fb79"} Jul 14 19:08:24.692224 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.691991 2569 scope.go:117] "RemoveContainer" containerID="fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a" Jul 14 19:08:24.700564 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.700544 2569 scope.go:117] "RemoveContainer" containerID="fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a" Jul 14 19:08:24.700827 ip-10-0-130-79 kubenswrapper[2569]: E0714 19:08:24.700811 2569 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a\": container with ID starting with fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a not found: ID does not exist" containerID="fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a" Jul 14 19:08:24.700870 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.700834 2569 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a"} err="failed to get container status \"fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a\": rpc error: code = NotFound desc = could not find container \"fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a\": container with ID starting with fefb9b9d9e2289aea00a02544104857fd96ad882766d5721da813d671b447e1a not found: ID does not exist" Jul 14 19:08:24.709793 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.709766 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d"] Jul 14 19:08:24.713564 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:24.713535 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-10fa01/rate-test-model-10fa01-kserve-755c795c87-c8j7d"] Jul 14 19:08:25.726027 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.725994 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:25.726434 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.726287 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3deb05fb-09ec-48be-8267-d13a85c6a1eb" containerName="main" Jul 14 19:08:25.726434 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.726297 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="3deb05fb-09ec-48be-8267-d13a85c6a1eb" containerName="main" Jul 14 19:08:25.726434 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.726309 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b231ee2f-006b-410f-9437-9f9281b5ea56" containerName="main" Jul 14 19:08:25.726434 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.726315 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="b231ee2f-006b-410f-9437-9f9281b5ea56" containerName="main" Jul 14 19:08:25.726434 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.726376 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="3deb05fb-09ec-48be-8267-d13a85c6a1eb" containerName="main" Jul 14 19:08:25.726434 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.726384 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="b231ee2f-006b-410f-9437-9f9281b5ea56" containerName="main" Jul 14 19:08:25.730629 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.730612 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:25.736977 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.736957 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-mqq4m\"" Jul 14 19:08:25.758031 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.758005 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:25.866525 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.866488 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2s7g\" (UniqueName: \"kubernetes.io/projected/a5158fd1-fc01-49da-a5b5-416da5f66a1b-kube-api-access-h2s7g\") pod \"test-curl-7944-139954953696288\" (UID: \"a5158fd1-fc01-49da-a5b5-416da5f66a1b\") " pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:25.967919 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.967883 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2s7g\" (UniqueName: \"kubernetes.io/projected/a5158fd1-fc01-49da-a5b5-416da5f66a1b-kube-api-access-h2s7g\") pod \"test-curl-7944-139954953696288\" (UID: \"a5158fd1-fc01-49da-a5b5-416da5f66a1b\") " pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:25.977781 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:25.977716 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2s7g\" (UniqueName: \"kubernetes.io/projected/a5158fd1-fc01-49da-a5b5-416da5f66a1b-kube-api-access-h2s7g\") pod \"test-curl-7944-139954953696288\" (UID: \"a5158fd1-fc01-49da-a5b5-416da5f66a1b\") " pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:26.038993 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:26.038955 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:26.169032 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:26.169003 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:26.171079 ip-10-0-130-79 kubenswrapper[2569]: W0714 19:08:26.171048 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda5158fd1_fc01_49da_a5b5_416da5f66a1b.slice/crio-ef19226cf959cda61d083dad88dd0b21d2dc492898b8d2a8597861ed31d7a4ab WatchSource:0}: Error finding container ef19226cf959cda61d083dad88dd0b21d2dc492898b8d2a8597861ed31d7a4ab: Status 404 returned error can't find the container with id ef19226cf959cda61d083dad88dd0b21d2dc492898b8d2a8597861ed31d7a4ab Jul 14 19:08:26.172314 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:26.172295 2569 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 14 19:08:26.483213 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:26.483182 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3deb05fb-09ec-48be-8267-d13a85c6a1eb" path="/var/lib/kubelet/pods/3deb05fb-09ec-48be-8267-d13a85c6a1eb/volumes" Jul 14 19:08:26.706449 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:26.706364 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953696288" event={"ID":"a5158fd1-fc01-49da-a5b5-416da5f66a1b","Type":"ContainerStarted","Data":"ef19226cf959cda61d083dad88dd0b21d2dc492898b8d2a8597861ed31d7a4ab"} Jul 14 19:08:27.713568 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:27.713531 2569 generic.go:356] "Generic (PLEG): container finished" podID="a5158fd1-fc01-49da-a5b5-416da5f66a1b" containerID="3da65db0f930e1b81c3d1d4020dbf34c4a8b4046c7e5eac8bfe3ca98c236af1b" exitCode=0 Jul 14 19:08:27.713956 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:27.713597 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7944-139954953696288" event={"ID":"a5158fd1-fc01-49da-a5b5-416da5f66a1b","Type":"ContainerDied","Data":"3da65db0f930e1b81c3d1d4020dbf34c4a8b4046c7e5eac8bfe3ca98c236af1b"} Jul 14 19:08:28.866589 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:28.866566 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:28.906964 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:28.906937 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7944-139954953696288_a5158fd1-fc01-49da-a5b5-416da5f66a1b/test-curl-7944-139954953696288/0.log" Jul 14 19:08:28.932700 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:28.932657 2569 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:28.938828 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:28.938801 2569 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7944-139954953696288"] Jul 14 19:08:28.995811 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:28.995728 2569 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2s7g\" (UniqueName: \"kubernetes.io/projected/a5158fd1-fc01-49da-a5b5-416da5f66a1b-kube-api-access-h2s7g\") pod \"a5158fd1-fc01-49da-a5b5-416da5f66a1b\" (UID: \"a5158fd1-fc01-49da-a5b5-416da5f66a1b\") " Jul 14 19:08:28.997894 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:28.997866 2569 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5158fd1-fc01-49da-a5b5-416da5f66a1b-kube-api-access-h2s7g" (OuterVolumeSpecName: "kube-api-access-h2s7g") pod "a5158fd1-fc01-49da-a5b5-416da5f66a1b" (UID: "a5158fd1-fc01-49da-a5b5-416da5f66a1b"). InnerVolumeSpecName "kube-api-access-h2s7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 14 19:08:29.096811 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:29.096777 2569 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2s7g\" (UniqueName: \"kubernetes.io/projected/a5158fd1-fc01-49da-a5b5-416da5f66a1b-kube-api-access-h2s7g\") on node \"ip-10-0-130-79.ec2.internal\" DevicePath \"\"" Jul 14 19:08:29.724502 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:29.724471 2569 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:29.724502 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:29.724506 2569 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef19226cf959cda61d083dad88dd0b21d2dc492898b8d2a8597861ed31d7a4ab" Jul 14 19:08:29.734494 ip-10-0-130-79 kubenswrapper[2569]: E0714 19:08:29.734464 2569 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7944-139954953696288\" is forbidden: User \"system:node:ip-10-0-130-79.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-130-79.ec2.internal' and this object" podUID="a5158fd1-fc01-49da-a5b5-416da5f66a1b" pod="openshift-ingress/test-curl-7944-139954953696288" Jul 14 19:08:30.481737 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:30.481707 2569 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5158fd1-fc01-49da-a5b5-416da5f66a1b" path="/var/lib/kubelet/pods/a5158fd1-fc01-49da-a5b5-416da5f66a1b/volumes" Jul 14 19:08:43.404830 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:08:43.404787 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:09:09.093444 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:09.093403 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-8cxq5"] Jul 14 19:09:21.926214 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:21.926182 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68797955b-nkq9q_90012bb3-7876-497e-a403-08c133191e2e/authorino/0.log" Jul 14 19:09:28.199310 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:28.199275 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-c9r9f_0ed3bfaf-da25-4d88-a142-a15ee0000cf1/copy-manifests/0.log" Jul 14 19:09:28.206712 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:28.206678 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-c9r9f_0ed3bfaf-da25-4d88-a142-a15ee0000cf1/manager/0.log" Jul 14 19:09:31.383038 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.383010 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp_3d2ab384-7cc6-4aa9-94b1-255c2c919320/util/0.log" Jul 14 19:09:31.390406 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.390385 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp_3d2ab384-7cc6-4aa9-94b1-255c2c919320/pull/0.log" Jul 14 19:09:31.398055 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.398033 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp_3d2ab384-7cc6-4aa9-94b1-255c2c919320/extract/0.log" Jul 14 19:09:31.517821 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.517792 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc_62a38bbb-557d-4a44-9442-0db22ca8bcd6/util/0.log" Jul 14 19:09:31.525236 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.525211 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc_62a38bbb-557d-4a44-9442-0db22ca8bcd6/pull/0.log" Jul 14 19:09:31.534987 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.534967 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc_62a38bbb-557d-4a44-9442-0db22ca8bcd6/extract/0.log" Jul 14 19:09:31.655301 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.655207 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx_61436e65-1689-4d0d-897c-e0285149edc6/util/0.log" Jul 14 19:09:31.663242 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.663206 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx_61436e65-1689-4d0d-897c-e0285149edc6/pull/0.log" Jul 14 19:09:31.672442 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.672422 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx_61436e65-1689-4d0d-897c-e0285149edc6/extract/0.log" Jul 14 19:09:31.800020 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.799986 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb_cc7a3d39-6b7a-4005-a8f3-20523507a713/util/0.log" Jul 14 19:09:31.807699 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.807653 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb_cc7a3d39-6b7a-4005-a8f3-20523507a713/pull/0.log" Jul 14 19:09:31.815732 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.815713 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb_cc7a3d39-6b7a-4005-a8f3-20523507a713/extract/0.log" Jul 14 19:09:31.950002 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:31.949907 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68797955b-nkq9q_90012bb3-7876-497e-a403-08c133191e2e/authorino/0.log" Jul 14 19:09:32.320056 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:32.320029 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-q5hv7_9a665233-2e99-427f-9197-bc5408c47dc2/kuadrant-console-plugin/0.log" Jul 14 19:09:32.714818 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:32.714742 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8cxq5_09ffa02e-e0d5-4ecb-98cb-5e4907efd905/limitador/0.log" Jul 14 19:09:33.508432 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:33.508397 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d44f677bf-jj4k7_ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2/kube-auth-proxy/0.log" Jul 14 19:09:35.048711 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:35.048684 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-kg25l_b569c6ab-9898-4f57-a4e2-94ece72a1a19/main/0.log" Jul 14 19:09:35.174322 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:35.174287 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-gvqsx_b37b2919-d842-46d2-98ea-b3c464ef6806/main/0.log" Jul 14 19:09:42.434747 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:42.434672 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-rx2bj_70bf487c-b158-4aff-9aa0-bc83b031e407/global-pull-secret-syncer/0.log" Jul 14 19:09:42.545610 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:42.545580 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9rm2m_2d4b6cc7-09ea-4477-9224-a59618f6e2d3/konnectivity-agent/0.log" Jul 14 19:09:42.610210 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:42.610181 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-79.ec2.internal_799b04e74e4d564ee24fa3864d962c91/haproxy/0.log" Jul 14 19:09:47.940564 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:47.940524 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp_3d2ab384-7cc6-4aa9-94b1-255c2c919320/extract/0.log" Jul 14 19:09:47.966321 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:47.966292 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp_3d2ab384-7cc6-4aa9-94b1-255c2c919320/util/0.log" Jul 14 19:09:47.990975 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:47.990949 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759g26pp_3d2ab384-7cc6-4aa9-94b1-255c2c919320/pull/0.log" Jul 14 19:09:48.035708 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.035671 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc_62a38bbb-557d-4a44-9442-0db22ca8bcd6/extract/0.log" Jul 14 19:09:48.088363 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.088333 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc_62a38bbb-557d-4a44-9442-0db22ca8bcd6/util/0.log" Jul 14 19:09:48.113535 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.113508 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e085cvc_62a38bbb-557d-4a44-9442-0db22ca8bcd6/pull/0.log" Jul 14 19:09:48.146066 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.146037 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx_61436e65-1689-4d0d-897c-e0285149edc6/extract/0.log" Jul 14 19:09:48.169337 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.169313 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx_61436e65-1689-4d0d-897c-e0285149edc6/util/0.log" Jul 14 19:09:48.194182 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.194071 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73l5wdx_61436e65-1689-4d0d-897c-e0285149edc6/pull/0.log" Jul 14 19:09:48.228603 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.228578 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb_cc7a3d39-6b7a-4005-a8f3-20523507a713/extract/0.log" Jul 14 19:09:48.254960 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.254935 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb_cc7a3d39-6b7a-4005-a8f3-20523507a713/util/0.log" Jul 14 19:09:48.281685 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.281654 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jjqkb_cc7a3d39-6b7a-4005-a8f3-20523507a713/pull/0.log" Jul 14 19:09:48.594884 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.594848 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-68797955b-nkq9q_90012bb3-7876-497e-a403-08c133191e2e/authorino/0.log" Jul 14 19:09:48.705413 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.705383 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-q5hv7_9a665233-2e99-427f-9197-bc5408c47dc2/kuadrant-console-plugin/0.log" Jul 14 19:09:48.964072 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:48.963995 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-8cxq5_09ffa02e-e0d5-4ecb-98cb-5e4907efd905/limitador/0.log" Jul 14 19:09:50.941808 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:50.941778 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-68ffd96d67-dmvng_23c985fb-f138-4dda-a2e2-1f2b70bc8889/metrics-server/0.log" Jul 14 19:09:51.220854 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:51.220777 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swldn_0edeb880-60bd-454a-ad60-4debf10076f1/node-exporter/0.log" Jul 14 19:09:51.250369 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:51.250345 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swldn_0edeb880-60bd-454a-ad60-4debf10076f1/kube-rbac-proxy/0.log" Jul 14 19:09:51.273367 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:51.273341 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-swldn_0edeb880-60bd-454a-ad60-4debf10076f1/init-textfile/0.log" Jul 14 19:09:51.619104 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:51.619075 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-x9qqw_a7285a78-51a1-4e85-bffe-9544e843dafb/prometheus-operator/0.log" Jul 14 19:09:51.641683 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:51.641658 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-x9qqw_a7285a78-51a1-4e85-bffe-9544e843dafb/kube-rbac-proxy/0.log" Jul 14 19:09:52.757681 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.757645 2569 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg"] Jul 14 19:09:52.758198 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.758101 2569 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5158fd1-fc01-49da-a5b5-416da5f66a1b" containerName="test-curl-7944-139954953696288" Jul 14 19:09:52.758198 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.758118 2569 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5158fd1-fc01-49da-a5b5-416da5f66a1b" containerName="test-curl-7944-139954953696288" Jul 14 19:09:52.758198 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.758169 2569 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5158fd1-fc01-49da-a5b5-416da5f66a1b" containerName="test-curl-7944-139954953696288" Jul 14 19:09:52.761387 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.761369 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.764456 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.764434 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26nj6\"/\"openshift-service-ca.crt\"" Jul 14 19:09:52.765820 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.765796 2569 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-26nj6\"/\"default-dockercfg-9br9n\"" Jul 14 19:09:52.765921 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.765833 2569 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-26nj6\"/\"kube-root-ca.crt\"" Jul 14 19:09:52.771848 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.771823 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg"] Jul 14 19:09:52.813172 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.813143 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-podres\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.813359 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.813186 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-proc\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.813359 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.813223 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhqnz\" (UniqueName: \"kubernetes.io/projected/8ce68161-b505-452a-a5e6-72c870700757-kube-api-access-xhqnz\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.813359 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.813276 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-sys\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.813359 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.813316 2569 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-lib-modules\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914109 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914069 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-sys\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914333 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914117 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-lib-modules\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914333 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914144 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-podres\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914333 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914173 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-proc\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914333 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914200 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-sys\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914333 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914213 2569 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhqnz\" (UniqueName: \"kubernetes.io/projected/8ce68161-b505-452a-a5e6-72c870700757-kube-api-access-xhqnz\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914333 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914322 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-proc\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914570 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914351 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-lib-modules\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.914570 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.914353 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/8ce68161-b505-452a-a5e6-72c870700757-podres\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:52.923993 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:52.923967 2569 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhqnz\" (UniqueName: \"kubernetes.io/projected/8ce68161-b505-452a-a5e6-72c870700757-kube-api-access-xhqnz\") pod \"perf-node-gather-daemonset-8pvlg\" (UID: \"8ce68161-b505-452a-a5e6-72c870700757\") " pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:53.026177 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:53.026099 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-9t6rt_343af867-9336-435a-8a69-2f3365f6798e/networking-console-plugin/0.log" Jul 14 19:09:53.072793 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:53.072761 2569 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:53.208737 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:53.208645 2569 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg"] Jul 14 19:09:53.211491 ip-10-0-130-79 kubenswrapper[2569]: W0714 19:09:53.211455 2569 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8ce68161_b505_452a_a5e6_72c870700757.slice/crio-9dd70a446afe8f88cf8190f85b14aaef293a22eadb0a906e269a0bcfcb7be507 WatchSource:0}: Error finding container 9dd70a446afe8f88cf8190f85b14aaef293a22eadb0a906e269a0bcfcb7be507: Status 404 returned error can't find the container with id 9dd70a446afe8f88cf8190f85b14aaef293a22eadb0a906e269a0bcfcb7be507 Jul 14 19:09:54.104708 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:54.104675 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-67449cbd47-2kmn9_16e9f4a2-15bf-46e5-93ed-9ddf19cdd87f/console/0.log" Jul 14 19:09:54.137950 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:54.137916 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" event={"ID":"8ce68161-b505-452a-a5e6-72c870700757","Type":"ContainerStarted","Data":"36b21e716487b93e6047a38fb19526afc238a50aaa780864180dd93b80845895"} Jul 14 19:09:54.138135 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:54.138008 2569 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" event={"ID":"8ce68161-b505-452a-a5e6-72c870700757","Type":"ContainerStarted","Data":"9dd70a446afe8f88cf8190f85b14aaef293a22eadb0a906e269a0bcfcb7be507"} Jul 14 19:09:54.138135 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:54.138038 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:09:54.158020 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:54.157972 2569 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" podStartSLOduration=2.157958326 podStartE2EDuration="2.157958326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 19:09:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 19:09:54.157642917 +0000 UTC m=+2888.215310987" watchObservedRunningTime="2026-07-14 19:09:54.157958326 +0000 UTC m=+2888.215626375" Jul 14 19:09:55.928026 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:55.927986 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hnl7l_5a06092a-5be1-49bf-98d3-4f47ae464faf/dns/0.log" Jul 14 19:09:55.951771 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:55.951720 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hnl7l_5a06092a-5be1-49bf-98d3-4f47ae464faf/kube-rbac-proxy/0.log" Jul 14 19:09:56.050764 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:56.050736 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-swtrt_11223d59-522e-4311-ad5c-fcc84cf423e8/dns-node-resolver/0.log" Jul 14 19:09:56.659965 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:56.659935 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-558f9b9449-ccgtp_e14c8b14-5897-48fb-8226-1d6ae22c52a1/registry/0.log" Jul 14 19:09:56.684726 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:56.684701 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-hbw2t_0a9e50d0-df10-47d4-9ad8-717bc1af9150/node-ca/0.log" Jul 14 19:09:58.033148 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:58.033117 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d44f677bf-jj4k7_ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2/kube-auth-proxy/0.log" Jul 14 19:09:59.122002 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:59.121976 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-z5d6b_0cf86795-02bf-4e52-a5ac-ae404d64f9bd/serve-healthcheck-canary/0.log" Jul 14 19:09:59.805720 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:59.805680 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l75ch_975f0279-7e84-48f1-932c-739390863380/kube-rbac-proxy/0.log" Jul 14 19:09:59.838003 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:59.837972 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l75ch_975f0279-7e84-48f1-932c-739390863380/exporter/0.log" Jul 14 19:09:59.869433 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:09:59.869406 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-l75ch_975f0279-7e84-48f1-932c-739390863380/extractor/0.log" Jul 14 19:10:00.153297 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:10:00.153208 2569 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-26nj6/perf-node-gather-daemonset-8pvlg" Jul 14 19:10:03.140577 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:10:03.140493 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d44f677bf-jj4k7_ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2/kube-auth-proxy/0.log" Jul 14 19:10:03.340596 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:10:03.340565 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-c9r9f_0ed3bfaf-da25-4d88-a142-a15ee0000cf1/manager/0.log" Jul 14 19:10:03.362358 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:10:03.362328 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5d948cf9bc-c9r9f_0ed3bfaf-da25-4d88-a142-a15ee0000cf1/copy-manifests/0.log" Jul 14 19:10:04.801342 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:10:04.801314 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5d44f677bf-jj4k7_ed27cfdc-d328-4b8e-a6e9-3b59b31c3ce2/kube-auth-proxy/0.log" Jul 14 19:10:04.808191 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:10:04.808169 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-8c5cc675c-m9p22_351c593d-6f03-4821-9527-b21eb1aefec1/manager/0.log" Jul 14 19:10:04.843105 ip-10-0-130-79 kubenswrapper[2569]: I0714 19:10:04.843073 2569 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-j5gcw_5ad1b22b-f53a-4ee5-bb47-0a6197a0cbd8/openshift-lws-operator/0.log"