Jul 21 21:46:20.391703 ip-10-0-138-36 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Jul 21 21:46:20.391712 ip-10-0-138-36 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Jul 21 21:46:20.391719 ip-10-0-138-36 systemd[1]: kubelet.service: Failed with result 'resources'. Jul 21 21:46:20.391936 ip-10-0-138-36 systemd[1]: Failed to start Kubernetes Kubelet. Jul 21 21:46:30.499261 ip-10-0-138-36 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found. Jul 21 21:46:30.499275 ip-10-0-138-36 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7e024ea14e7a4156883ba01e0f8dbc71 -- Jul 21 21:48:19.795433 ip-10-0-138-36 systemd[1]: Starting Kubernetes Kubelet... Jul 21 21:48:20.121584 ip-10-0-138-36 kubenswrapper[2557]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:20.121584 ip-10-0-138-36 kubenswrapper[2557]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 21 21:48:20.121584 ip-10-0-138-36 kubenswrapper[2557]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:20.121584 ip-10-0-138-36 kubenswrapper[2557]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 21 21:48:20.121584 ip-10-0-138-36 kubenswrapper[2557]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 21 21:48:20.122392 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.122300 2557 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 21 21:48:20.127618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127599 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127620 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127624 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127628 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127631 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127634 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127637 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127643 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127646 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127649 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127651 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127654 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127657 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127659 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127662 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127664 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127667 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:20.127663 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127671 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127674 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127677 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127679 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127682 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127685 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127687 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127690 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127693 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127696 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127698 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127701 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127703 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127706 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127708 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127711 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127713 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127716 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127720 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127724 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:20.128133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127727 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127730 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127733 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127735 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127738 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127741 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127743 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127746 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127748 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127750 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127753 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127755 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127758 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127760 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127765 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127767 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127770 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127772 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127775 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127777 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:20.128619 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127780 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127783 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127785 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127788 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127790 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127792 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127795 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127797 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127800 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127804 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127807 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127810 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127813 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127817 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127820 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127823 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127826 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127829 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127831 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:20.129126 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127834 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127836 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127839 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127842 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127844 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127847 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127849 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127853 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127855 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127860 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127864 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127867 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127869 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127872 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127875 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127877 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127880 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127882 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127885 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127888 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:20.129582 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127891 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127893 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.127896 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128647 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128654 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128657 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128660 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128663 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128666 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128671 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128675 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128678 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128681 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128684 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128686 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128690 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128692 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128695 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128699 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:20.130065 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128702 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128705 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128708 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128710 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128713 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128715 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128718 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128721 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128723 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128725 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128728 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128730 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128733 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128735 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128737 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128740 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128742 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128745 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128747 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128749 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:20.130591 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128752 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128754 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128757 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128760 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128762 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128765 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128767 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128770 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128772 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128775 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128777 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128780 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128783 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128786 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128788 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128790 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128793 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128796 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128798 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:20.131150 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128801 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128803 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128805 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128808 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128810 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128813 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128815 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128818 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128822 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128826 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128830 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128832 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128834 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128837 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128839 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128842 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128845 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128847 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128849 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128852 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:20.131618 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128854 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128857 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128859 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128861 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128864 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128867 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128872 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128875 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128878 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128881 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128884 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128886 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128889 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128892 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128895 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128897 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128900 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128903 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128906 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:20.132121 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128909 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128911 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128914 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128916 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.128919 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129334 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129340 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129343 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129346 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129350 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129353 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129355 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129358 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129361 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129363 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129367 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129370 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129373 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129376 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129380 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:20.132616 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129384 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129387 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129390 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129393 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129395 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129397 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129400 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129403 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129405 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129408 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129410 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129413 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129415 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129418 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129420 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129423 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129425 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129428 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129431 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:20.133113 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129433 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129436 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129438 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129441 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129444 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129446 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129449 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129451 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129453 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129456 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129459 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129464 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129467 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129470 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129472 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129475 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129477 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129479 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129482 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129484 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:20.133583 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129486 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129489 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129491 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129493 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129496 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129498 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129502 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129506 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129509 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129512 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129514 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129517 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129520 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129522 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129525 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129528 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129531 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129533 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129535 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:20.134070 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129538 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129540 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129543 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129545 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129550 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129552 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129555 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129558 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129560 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129563 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129565 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129567 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129570 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129572 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129575 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129577 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129580 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129582 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129585 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:20.134543 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129588 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129590 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129593 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129595 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129598 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129600 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.129603 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129673 2557 flags.go:64] FLAG: --address="0.0.0.0" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129683 2557 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129690 2557 flags.go:64] FLAG: --anonymous-auth="true" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129696 2557 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129703 2557 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129708 2557 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129714 2557 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129721 2557 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129725 2557 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129727 2557 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129732 2557 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129735 2557 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129740 2557 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129743 2557 flags.go:64] FLAG: --cgroup-root="" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129746 2557 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129749 2557 flags.go:64] FLAG: --client-ca-file="" Jul 21 21:48:20.135008 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129752 2557 flags.go:64] FLAG: --cloud-provider="external" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129755 2557 flags.go:64] FLAG: --cluster-dns="[]" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129759 2557 flags.go:64] FLAG: --cluster-domain="" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129762 2557 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129765 2557 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129768 2557 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129771 2557 flags.go:64] FLAG: --container-log-max-files="5" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129775 2557 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129778 2557 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129781 2557 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129784 2557 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129787 2557 flags.go:64] FLAG: --contention-profiling="false" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129790 2557 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129793 2557 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129796 2557 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129799 2557 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129803 2557 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129807 2557 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129811 2557 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129814 2557 flags.go:64] FLAG: --enable-load-reader="false" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129817 2557 flags.go:64] FLAG: --enable-server="true" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129821 2557 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129825 2557 flags.go:64] FLAG: --event-burst="100" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129828 2557 flags.go:64] FLAG: --event-qps="50" Jul 21 21:48:20.135574 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129831 2557 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129834 2557 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129839 2557 flags.go:64] FLAG: --eviction-hard="" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129843 2557 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129845 2557 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129848 2557 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129851 2557 flags.go:64] FLAG: --eviction-soft="" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129854 2557 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129857 2557 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129860 2557 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129863 2557 flags.go:64] FLAG: --experimental-mounter-path="" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129866 2557 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129869 2557 flags.go:64] FLAG: --fail-swap-on="true" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129871 2557 flags.go:64] FLAG: --feature-gates="" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129875 2557 flags.go:64] FLAG: --file-check-frequency="20s" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129878 2557 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129881 2557 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129884 2557 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129887 2557 flags.go:64] FLAG: --healthz-port="10248" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129890 2557 flags.go:64] FLAG: --help="false" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129893 2557 flags.go:64] FLAG: --hostname-override="ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129896 2557 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129899 2557 flags.go:64] FLAG: --http-check-frequency="20s" Jul 21 21:48:20.136217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129902 2557 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129906 2557 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129910 2557 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129913 2557 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129917 2557 flags.go:64] FLAG: --image-service-endpoint="" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129920 2557 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129923 2557 flags.go:64] FLAG: --kube-api-burst="100" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129926 2557 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129929 2557 flags.go:64] FLAG: --kube-api-qps="50" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129932 2557 flags.go:64] FLAG: --kube-reserved="" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129935 2557 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129940 2557 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129943 2557 flags.go:64] FLAG: --kubelet-cgroups="" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129946 2557 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129949 2557 flags.go:64] FLAG: --lock-file="" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129952 2557 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129955 2557 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129958 2557 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129963 2557 flags.go:64] FLAG: --log-json-split-stream="false" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129966 2557 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129969 2557 flags.go:64] FLAG: --log-text-split-stream="false" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129972 2557 flags.go:64] FLAG: --logging-format="text" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129975 2557 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129978 2557 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 21 21:48:20.136885 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129981 2557 flags.go:64] FLAG: --manifest-url="" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129984 2557 flags.go:64] FLAG: --manifest-url-header="" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129988 2557 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129991 2557 flags.go:64] FLAG: --max-open-files="1000000" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129995 2557 flags.go:64] FLAG: --max-pods="110" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.129998 2557 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130001 2557 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130004 2557 flags.go:64] FLAG: --memory-manager-policy="None" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130007 2557 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130010 2557 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130014 2557 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130017 2557 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130031 2557 flags.go:64] FLAG: --node-status-max-images="50" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130034 2557 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130037 2557 flags.go:64] FLAG: --oom-score-adj="-999" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130040 2557 flags.go:64] FLAG: --pod-cidr="" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130043 2557 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130050 2557 flags.go:64] FLAG: --pod-manifest-path="" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130053 2557 flags.go:64] FLAG: --pod-max-pids="-1" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130058 2557 flags.go:64] FLAG: --pods-per-core="0" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130061 2557 flags.go:64] FLAG: --port="10250" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130064 2557 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130067 2557 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02d3f4a67ef36909d" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130070 2557 flags.go:64] FLAG: --qos-reserved="" Jul 21 21:48:20.137927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130073 2557 flags.go:64] FLAG: --read-only-port="10255" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130076 2557 flags.go:64] FLAG: --register-node="true" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130079 2557 flags.go:64] FLAG: --register-with-taints="" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130082 2557 flags.go:64] FLAG: --registry-burst="10" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130085 2557 flags.go:64] FLAG: --registry-qps="5" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130101 2557 flags.go:64] FLAG: --reserved-cpus="" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130104 2557 flags.go:64] FLAG: --reserved-memory="" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130107 2557 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130110 2557 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130113 2557 flags.go:64] FLAG: --rotate-certificates="false" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130117 2557 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130119 2557 flags.go:64] FLAG: --runonce="false" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130122 2557 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130125 2557 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130128 2557 flags.go:64] FLAG: --seccomp-default="false" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130131 2557 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130134 2557 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130137 2557 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130140 2557 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130144 2557 flags.go:64] FLAG: --storage-driver-password="root" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130147 2557 flags.go:64] FLAG: --storage-driver-secure="false" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130151 2557 flags.go:64] FLAG: --storage-driver-table="stats" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130153 2557 flags.go:64] FLAG: --storage-driver-user="root" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130156 2557 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130159 2557 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 21 21:48:20.138967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130164 2557 flags.go:64] FLAG: --system-cgroups="" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130167 2557 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130174 2557 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130177 2557 flags.go:64] FLAG: --tls-cert-file="" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130180 2557 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130184 2557 flags.go:64] FLAG: --tls-min-version="" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130187 2557 flags.go:64] FLAG: --tls-private-key-file="" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130190 2557 flags.go:64] FLAG: --topology-manager-policy="none" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130193 2557 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130196 2557 flags.go:64] FLAG: --topology-manager-scope="container" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130198 2557 flags.go:64] FLAG: --v="2" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130203 2557 flags.go:64] FLAG: --version="false" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130207 2557 flags.go:64] FLAG: --vmodule="" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130210 2557 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.130229 2557 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130348 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130352 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130355 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130358 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130361 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130364 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130366 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130368 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:20.140062 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130373 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130376 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130379 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130384 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130387 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130390 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130393 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130395 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130398 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130401 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130405 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130409 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130412 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130414 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130417 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130419 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130422 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130424 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130427 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130431 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:20.141072 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130434 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130437 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130440 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130443 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130446 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130448 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130451 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130453 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130455 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130458 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130460 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130463 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130465 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130467 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130470 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130473 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130476 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130478 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130481 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130483 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:20.141720 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130486 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130488 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130491 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130496 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130498 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130501 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130503 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130506 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130508 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130511 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130513 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130515 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130518 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130520 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130522 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130525 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130527 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130530 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130532 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130534 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:20.142559 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130537 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130539 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130542 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130544 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130546 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130550 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130554 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130557 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130560 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130563 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130566 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130569 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130571 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130574 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130577 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130580 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130586 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130588 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130591 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:20.143395 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130593 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130596 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130599 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130602 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130604 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130607 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130609 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130612 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130614 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130617 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130619 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.130622 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.131186 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.139407 2557 server.go:529] "Kubelet version" kubeletVersion="v1.34.9" Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.139432 2557 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 21 21:48:20.143958 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.139460 2557 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.139470 2557 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139567 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139576 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139581 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139585 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139589 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139594 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139598 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139602 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139607 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139611 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139616 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139620 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139624 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139628 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139633 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139637 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:20.144570 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139641 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139647 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139653 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139658 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139662 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139667 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139671 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139675 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139679 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139683 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139687 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139691 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139695 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139708 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139713 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139718 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139722 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139726 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139730 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139734 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:20.145037 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139738 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139742 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139747 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139751 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139755 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139759 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139762 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139767 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139771 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139775 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139780 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139784 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139788 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139792 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139796 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139800 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139805 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139809 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139813 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139817 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:20.145744 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139822 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139826 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139830 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139834 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139838 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139844 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139856 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139861 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139866 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139869 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139874 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139878 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139882 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139886 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139890 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139894 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139898 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139905 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139911 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139916 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:20.146335 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139920 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139925 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139930 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139934 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139938 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139945 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139950 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139955 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139959 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139964 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139968 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139972 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139975 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139980 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139984 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139988 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139992 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.139997 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140001 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140012 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:20.146851 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140017 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140021 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140025 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.140033 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140282 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140301 2557 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140306 2557 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140311 2557 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140316 2557 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140321 2557 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140325 2557 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140329 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140333 2557 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140338 2557 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140342 2557 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 21 21:48:20.147434 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140346 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140350 2557 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140354 2557 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140358 2557 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140362 2557 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140366 2557 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140370 2557 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140375 2557 feature_gate.go:328] unrecognized feature gate: Example2 Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140379 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140383 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140387 2557 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140391 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140395 2557 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140399 2557 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140405 2557 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140409 2557 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140413 2557 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140425 2557 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140429 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140433 2557 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 21 21:48:20.147866 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140437 2557 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140442 2557 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140446 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140450 2557 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140454 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140459 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140463 2557 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140467 2557 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140471 2557 feature_gate.go:328] unrecognized feature gate: Example Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140475 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140479 2557 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140483 2557 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140487 2557 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140492 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140496 2557 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140500 2557 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140504 2557 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140508 2557 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140514 2557 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 21 21:48:20.148406 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140520 2557 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140525 2557 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140531 2557 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140536 2557 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140541 2557 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140545 2557 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140551 2557 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140556 2557 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140560 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140566 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140570 2557 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140582 2557 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140587 2557 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140591 2557 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140595 2557 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140599 2557 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140604 2557 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140608 2557 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140612 2557 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 21 21:48:20.149214 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140616 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140623 2557 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140628 2557 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140633 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140637 2557 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140641 2557 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140646 2557 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140651 2557 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140656 2557 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140660 2557 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140664 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140668 2557 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140673 2557 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140678 2557 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140682 2557 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140687 2557 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140692 2557 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140696 2557 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140700 2557 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140705 2557 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 21 21:48:20.149801 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140709 2557 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140714 2557 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140718 2557 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140722 2557 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140726 2557 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140739 2557 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140744 2557 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140748 2557 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140752 2557 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:20.140756 2557 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.140765 2557 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.140993 2557 server.go:962] "Client rotation is on, will bootstrap in background" Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.144183 2557 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.146225 2557 server.go:1019] "Starting client certificate rotation" Jul 21 21:48:20.150493 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.146339 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:48:20.150920 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.146968 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 21 21:48:20.150920 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.148524 2557 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:48:20.152451 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.152431 2557 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 21 21:48:20.161249 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.161230 2557 log.go:25] "Validated CRI v1 runtime API" Jul 21 21:48:20.164959 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.164944 2557 log.go:25] "Validated CRI v1 image API" Jul 21 21:48:20.166257 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.166235 2557 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 21 21:48:20.169708 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.169682 2557 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a58786e4-0b06-4110-8261-ff79ca0f738d:/dev/nvme0n1p3 dc9eb191-90ac-42b1-93bf-ae1f2b10eb65:/dev/nvme0n1p4] Jul 21 21:48:20.169778 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.169707 2557 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 21 21:48:20.174533 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.174419 2557 manager.go:216] Machine: {Timestamp:2026-07-21 21:48:20.173912097 +0000 UTC m=+0.279876254 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3097680 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec226451054da2d84433f53e88c21f86 SystemUUID:ec226451-054d-a2d8-4433-f53e88c21f86 BootID:7e024ea1-4e7a-4156-883b-a01e0f8dbc71 Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:7a:1d:9f:7a:77 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:7a:1d:9f:7a:77 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:94:61:23:11:49 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 21 21:48:20.174533 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.174528 2557 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 21 21:48:20.174639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.174608 2557 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 21 21:48:20.177374 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.177342 2557 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 21 21:48:20.177512 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.177378 2557 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-36.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 21 21:48:20.177988 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.177978 2557 topology_manager.go:138] "Creating topology manager with none policy" Jul 21 21:48:20.178024 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.177992 2557 container_manager_linux.go:309] "Creating device plugin manager" Jul 21 21:48:20.178024 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.177999 2557 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:48:20.178576 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.178560 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:48:20.178698 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.178686 2557 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 21 21:48:20.179946 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.179936 2557 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 21 21:48:20.184114 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184086 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:48:20.184242 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184233 2557 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 21 21:48:20.184278 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184268 2557 kubelet.go:486] "Attempting to sync node with API server" Jul 21 21:48:20.184322 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184311 2557 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 21 21:48:20.184352 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184332 2557 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 21 21:48:20.184352 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184346 2557 kubelet.go:398] "Adding apiserver pod source" Jul 21 21:48:20.184409 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184358 2557 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 21 21:48:20.184812 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184801 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:48:20.184848 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.184820 2557 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 21 21:48:20.186951 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.186932 2557 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1" Jul 21 21:48:20.189950 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.189932 2557 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 21 21:48:20.190018 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.189965 2557 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 21 21:48:20.191005 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.190991 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 21 21:48:20.191053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191010 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 21 21:48:20.191053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191018 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 21 21:48:20.191053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191024 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 21 21:48:20.191053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191032 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 21 21:48:20.191053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191038 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 21 21:48:20.191053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191046 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 21 21:48:20.191053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191051 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 21 21:48:20.191271 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191058 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 21 21:48:20.191271 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191063 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 21 21:48:20.191271 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191072 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 21 21:48:20.191271 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191077 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 21 21:48:20.191687 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191678 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 21 21:48:20.191687 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.191687 2557 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 21 21:48:20.194929 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.194910 2557 server.go:1268] "Started kubelet" Jul 21 21:48:20.195025 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.194972 2557 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-36.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 21 21:48:20.195025 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.194979 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 21 21:48:20.195156 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.195025 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 21 21:48:20.195156 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.195036 2557 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 21 21:48:20.195156 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.195048 2557 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 21 21:48:20.195156 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.195077 2557 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 21 21:48:20.195845 ip-10-0-138-36 systemd[1]: Started Kubernetes Kubelet. Jul 21 21:48:20.196826 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.196671 2557 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 21 21:48:20.196892 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.196877 2557 server.go:310] "Adding debug handlers to kubelet server" Jul 21 21:48:20.200895 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.200866 2557 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 21 21:48:20.202784 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.202010 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-36.ec2.internal.18c46c5d453c21f0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-36.ec2.internal,UID:ip-10-0-138-36.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-36.ec2.internal,},FirstTimestamp:2026-07-21 21:48:20.194886128 +0000 UTC m=+0.300850266,LastTimestamp:2026-07-21 21:48:20.194886128 +0000 UTC m=+0.300850266,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-36.ec2.internal,}" Jul 21 21:48:20.204025 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.204010 2557 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 21 21:48:20.204025 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.204019 2557 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 21 21:48:20.204162 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.204136 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:20.204622 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.204601 2557 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 21 21:48:20.204872 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.204855 2557 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 21 21:48:20.204872 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.204873 2557 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 21 21:48:20.205020 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.204975 2557 factory.go:55] Registering systemd factory Jul 21 21:48:20.205020 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.204986 2557 reconstruct.go:97] "Volume reconstruction finished" Jul 21 21:48:20.205020 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.204994 2557 reconciler.go:29] "Reconciler: start to sync state" Jul 21 21:48:20.205020 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.205000 2557 factory.go:223] Registration of the systemd container factory successfully Jul 21 21:48:20.205374 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.205268 2557 factory.go:153] Registering CRI-O factory Jul 21 21:48:20.205374 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.205280 2557 factory.go:223] Registration of the crio container factory successfully Jul 21 21:48:20.205374 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.205322 2557 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 21 21:48:20.205374 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.205345 2557 factory.go:103] Registering Raw factory Jul 21 21:48:20.205374 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.205359 2557 manager.go:1196] Started watching for new ooms in manager Jul 21 21:48:20.205966 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.205950 2557 manager.go:349] Starting recovery of all containers Jul 21 21:48:20.206712 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.206602 2557 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-36.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 21 21:48:20.206802 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.206762 2557 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 21 21:48:20.217784 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.217760 2557 manager.go:354] Recovery completed Jul 21 21:48:20.223242 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.223129 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:20.224675 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.224655 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lx67f" Jul 21 21:48:20.225131 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.225110 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:20.225203 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.225146 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:20.225203 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.225161 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:20.225636 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.225619 2557 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 21 21:48:20.225636 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.225633 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:20.225636 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.225636 2557 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 21 21:48:20.225797 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.225652 2557 state_mem.go:36] "Initialized new in-memory state store" Jul 21 21:48:20.226294 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.226281 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:20.226333 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.226303 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:20.226333 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.226313 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:20.227549 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.227484 2557 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-36.ec2.internal.18c46c5d4709a1e3 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-36.ec2.internal,UID:ip-10-0-138-36.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-36.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-36.ec2.internal,},FirstTimestamp:2026-07-21 21:48:20.225130979 +0000 UTC m=+0.331095119,LastTimestamp:2026-07-21 21:48:20.225130979 +0000 UTC m=+0.331095119,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-36.ec2.internal,}" Jul 21 21:48:20.227766 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.227754 2557 policy_none.go:49] "None policy: Start" Jul 21 21:48:20.227807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.227771 2557 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 21 21:48:20.227807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.227781 2557 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 21 21:48:20.229715 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.229705 2557 policy_none.go:47] "Start" Jul 21 21:48:20.230304 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.230289 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lx67f" Jul 21 21:48:20.264729 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.264709 2557 manager.go:337] "Starting Device Plugin manager" Jul 21 21:48:20.264845 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.264751 2557 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 21 21:48:20.264845 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.264764 2557 server.go:87] "Starting device plugin registration server" Jul 21 21:48:20.265061 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.265047 2557 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 21 21:48:20.265152 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.265064 2557 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 21 21:48:20.265203 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.265164 2557 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 21 21:48:20.265248 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.265241 2557 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 21 21:48:20.265306 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.265249 2557 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 21 21:48:20.265878 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.265858 2557 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 21 21:48:20.265970 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.265916 2557 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:20.365281 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.365245 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:20.367463 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.366220 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:20.367463 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.366258 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:20.367463 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.366270 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:20.367463 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.366297 2557 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.373432 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.373371 2557 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.373432 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.373399 2557 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-36.ec2.internal\": node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:20.381995 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.381972 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 21 21:48:20.383242 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.383222 2557 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 21 21:48:20.383350 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.383269 2557 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 21 21:48:20.383350 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.383298 2557 kubelet.go:2448] "Starting kubelet main sync loop" Jul 21 21:48:20.383350 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.383337 2557 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 21 21:48:20.385363 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.385345 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:20.402507 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.402479 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:20.484199 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.484147 2557 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal"] Jul 21 21:48:20.484383 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.484241 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:20.485266 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.485249 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:20.485372 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.485284 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:20.485372 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.485295 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:20.487602 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.487588 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:20.487728 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.487711 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.487769 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.487748 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:20.488339 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.488324 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:20.488417 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.488351 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:20.488417 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.488363 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:20.488914 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.488897 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:20.488998 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.488927 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:20.488998 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.488941 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:20.491176 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.491160 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.491260 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.491190 2557 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 21 21:48:20.491878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.491857 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientMemory" Jul 21 21:48:20.491967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.491888 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasNoDiskPressure" Jul 21 21:48:20.491967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.491902 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeHasSufficientPID" Jul 21 21:48:20.502596 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.502577 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:20.506962 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.506943 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a81050eca56907ec38c411d5d99cf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal\" (UID: \"06a81050eca56907ec38c411d5d99cf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.507052 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.506976 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e5af688db312735faccb6800239fbc06-config\") pod \"kube-apiserver-proxy-ip-10-0-138-36.ec2.internal\" (UID: \"e5af688db312735faccb6800239fbc06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.507052 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.507016 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a81050eca56907ec38c411d5d99cf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal\" (UID: \"06a81050eca56907ec38c411d5d99cf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.512996 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.512978 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-36.ec2.internal\" not found" node="ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.517524 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.517510 2557 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-36.ec2.internal\" not found" node="ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.603555 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.603509 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:20.607833 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.607815 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a81050eca56907ec38c411d5d99cf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal\" (UID: \"06a81050eca56907ec38c411d5d99cf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.607890 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.607841 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a81050eca56907ec38c411d5d99cf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal\" (UID: \"06a81050eca56907ec38c411d5d99cf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.607890 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.607858 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e5af688db312735faccb6800239fbc06-config\") pod \"kube-apiserver-proxy-ip-10-0-138-36.ec2.internal\" (UID: \"e5af688db312735faccb6800239fbc06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.607890 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.607884 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e5af688db312735faccb6800239fbc06-config\") pod \"kube-apiserver-proxy-ip-10-0-138-36.ec2.internal\" (UID: \"e5af688db312735faccb6800239fbc06\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.607984 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.607902 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/06a81050eca56907ec38c411d5d99cf2-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal\" (UID: \"06a81050eca56907ec38c411d5d99cf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.607984 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.607930 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/06a81050eca56907ec38c411d5d99cf2-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal\" (UID: \"06a81050eca56907ec38c411d5d99cf2\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.704330 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.704257 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:20.804742 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.804703 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:20.816032 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.816012 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.821069 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:20.820916 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" Jul 21 21:48:20.905687 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:20.905647 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:21.006152 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:21.006065 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:21.106545 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:21.106514 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:21.145315 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.145262 2557 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 21 21:48:21.145905 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.145504 2557 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 21 21:48:21.206605 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:21.206572 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:21.235411 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.235360 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 21:43:21 +0000 UTC" deadline="2028-02-28 18:15:39.294790246 +0000 UTC" Jul 21 21:48:21.235411 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.235403 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14084h27m18.059392596s" Jul 21 21:48:21.307543 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:21.307467 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:21.408631 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:21.408598 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:21.509385 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:21.509349 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:21.562561 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.562484 2557 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:21.576771 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:21.576730 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5af688db312735faccb6800239fbc06.slice/crio-80db9cee3e3c12b2e892ee31c6b671f5fb1cabc2a1c2abb7d8b34a15fa0e6fbb WatchSource:0}: Error finding container 80db9cee3e3c12b2e892ee31c6b671f5fb1cabc2a1c2abb7d8b34a15fa0e6fbb: Status 404 returned error can't find the container with id 80db9cee3e3c12b2e892ee31c6b671f5fb1cabc2a1c2abb7d8b34a15fa0e6fbb Jul 21 21:48:21.584287 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.584253 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:48:21.592649 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:21.592603 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06a81050eca56907ec38c411d5d99cf2.slice/crio-1a4e396189f6dc4e71e6dad4992352a170de3e7fb284720d6a8475c5b8170e24 WatchSource:0}: Error finding container 1a4e396189f6dc4e71e6dad4992352a170de3e7fb284720d6a8475c5b8170e24: Status 404 returned error can't find the container with id 1a4e396189f6dc4e71e6dad4992352a170de3e7fb284720d6a8475c5b8170e24 Jul 21 21:48:21.610137 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:21.610109 2557 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-36.ec2.internal\" not found" Jul 21 21:48:21.637660 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.637633 2557 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:21.699550 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.699523 2557 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:21.704931 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.704910 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" Jul 21 21:48:21.716658 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.716636 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:48:21.717593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.717580 2557 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" Jul 21 21:48:21.731764 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:21.731741 2557 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 21 21:48:22.187900 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.187869 2557 apiserver.go:52] "Watching apiserver" Jul 21 21:48:22.200840 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.200743 2557 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 21 21:48:22.202677 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.202625 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-szw9m","kube-system/konnectivity-agent-rbwsp","kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8","openshift-cluster-node-tuning-operator/tuned-nlfmw","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal","openshift-multus/multus-cgt88","openshift-network-operator/iptables-alerter-bqcmq","openshift-dns/node-resolver-gptzn","openshift-image-registry/node-ca-cdjcd","openshift-multus/multus-additional-cni-plugins-9xcsz","openshift-multus/network-metrics-daemon-9fdk4","openshift-network-diagnostics/network-check-target-hlh6j","openshift-ovn-kubernetes/ovnkube-node-z9qc4"] Jul 21 21:48:22.205881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.205294 2557 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 21 21:48:22.205881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.205388 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.208898 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.208384 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 21 21:48:22.208898 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.208410 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 21 21:48:22.208898 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.208440 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 21 21:48:22.208898 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.208473 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 21 21:48:22.208898 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.208598 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-vmdzp\"" Jul 21 21:48:22.208898 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.208717 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:22.208898 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.208791 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:22.211430 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.211405 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.213761 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.213736 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.216211 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.216155 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-pjv9w\"" Jul 21 21:48:22.216613 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.216593 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 21 21:48:22.216830 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.216809 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.217035 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.217233 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.217407 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.217593 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.218255 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-d856x\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.218276 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.218394 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.218653 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.218833 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219187 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-cni-binary-copy\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219221 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-netns\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219244 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-cni-bin\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219261 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-cni-multus\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219279 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-conf-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219296 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvrvq\" (UniqueName: \"kubernetes.io/projected/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-kube-api-access-zvrvq\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219326 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-socket-dir-parent\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219351 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-multus-certs\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219385 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-system-cni-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219432 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-cni-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219462 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-hostroot\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219509 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-cnibin\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.219881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219535 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.220883 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219539 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-os-release\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.220883 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219565 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-k8s-cni-cncf-io\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.220883 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219611 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-daemon-config\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.220883 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219643 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-etc-kubernetes\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.220883 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:22.220883 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.219689 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-kubelet\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.220883 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.220323 2557 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 21 21:48:22.221622 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.221490 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f84mv\"" Jul 21 21:48:22.221622 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.221558 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.222066 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.222032 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 21 21:48:22.222646 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.222508 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 21 21:48:22.222646 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.222520 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:48:22.222797 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.222723 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-blt9n\"" Jul 21 21:48:22.222883 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.222865 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:48:22.222990 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.222926 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 21 21:48:22.223861 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.223844 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 21 21:48:22.224240 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.224220 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 21 21:48:22.224550 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.224529 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4twls\"" Jul 21 21:48:22.224914 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.224896 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.224983 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.224965 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:22.227512 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.227486 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:22.229633 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.229614 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.229837 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.229814 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 21 21:48:22.230067 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.230033 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-r87v9\"" Jul 21 21:48:22.230223 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.230207 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 21 21:48:22.235706 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.235686 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 21 21:48:22.235921 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.235903 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 21 21:48:22.236191 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.236153 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-8x72r\"" Jul 21 21:48:22.236334 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.236314 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 21 21:48:22.236591 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.236571 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.239926 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.239904 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 21 21:48:22.240029 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.239946 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 21 21:48:22.240479 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.240442 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-9qb2v\"" Jul 21 21:48:22.240538 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.240482 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:22.240575 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.240539 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:22.240639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.240619 2557 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbgtl" Jul 21 21:48:22.251557 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.251532 2557 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-cbgtl" Jul 21 21:48:22.306344 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.306311 2557 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 21 21:48:22.320250 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320210 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-os-release\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.320391 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320260 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-daemon-config\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.320391 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:22.320391 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320326 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-systemd\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.320391 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320361 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.320391 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320388 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-cni-multus\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320454 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zswj\" (UniqueName: \"kubernetes.io/projected/d9c99127-500f-425b-91ce-a6472115be95-kube-api-access-9zswj\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320481 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ftl4\" (UniqueName: \"kubernetes.io/projected/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-kube-api-access-4ftl4\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320509 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-multus-certs\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320535 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320554 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysctl-conf\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-sys\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320596 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c22209d7-ce32-4882-870d-1ec2f71fe97a-dbus\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.320621 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320622 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sv6lp\" (UniqueName: \"kubernetes.io/projected/4eef30ff-e679-4d2e-bf2b-121e202bf436-kube-api-access-sv6lp\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320638 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9c99127-500f-425b-91ce-a6472115be95-host-slash\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e43a69f9-df08-4cfc-9704-a153830b131e-konnectivity-ca\") pod \"konnectivity-agent-rbwsp\" (UID: \"e43a69f9-df08-4cfc-9704-a153830b131e\") " pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-cni-netd\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320712 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-etc-kubernetes\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320728 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-registration-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320747 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-etc-selinux\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320763 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-conf-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320780 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9c99127-500f-425b-91ce-a6472115be95-iptables-alerter-script\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320805 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c22209d7-ce32-4882-870d-1ec2f71fe97a-kubelet-config\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320829 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-env-overrides\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-socket-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320869 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-run\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320886 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-socket-dir-parent\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320910 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03fb8246-7a76-492f-9e8b-b658d2a5fd49-tmp-dir\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.320927 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320927 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320944 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-device-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.320968 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-host\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321000 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321047 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-etc-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321077 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-cnibin\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-cni-binary-copy\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321130 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-kubelet\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321161 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-cni-binary-copy\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321202 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-netns\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321220 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovnkube-config\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321235 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-modprobe-d\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321275 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvqvg\" (UniqueName: \"kubernetes.io/projected/418b4e01-6bac-4840-8164-88a4f0ef298b-kube-api-access-xvqvg\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321327 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-kubelet\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-conf-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321362 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-systemd\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321393 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-os-release\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.321521 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321426 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321448 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-ovn\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321471 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-system-cni-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321475 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-socket-dir-parent\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321490 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-hostroot\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321508 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03fb8246-7a76-492f-9e8b-b658d2a5fd49-hosts-file\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321535 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e43a69f9-df08-4cfc-9704-a153830b131e-agent-certs\") pod \"konnectivity-agent-rbwsp\" (UID: \"e43a69f9-df08-4cfc-9704-a153830b131e\") " pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321556 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-k8s-cni-cncf-io\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321575 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/418b4e01-6bac-4840-8164-88a4f0ef298b-host\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321603 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321603 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-cnibin\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321620 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-var-lib-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321637 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-system-cni-dir\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321656 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4k9sv\" (UniqueName: \"kubernetes.io/projected/6907d1d1-462d-4594-ba2f-7c101f2f30fe-kube-api-access-4k9sv\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321661 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-kubelet\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321671 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pc8bq\" (UniqueName: \"kubernetes.io/projected/2b46dedd-8359-48ea-9a02-784534fcc254-kube-api-access-pc8bq\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321696 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysconfig\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.322243 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321724 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-cni-bin\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321752 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvrvq\" (UniqueName: \"kubernetes.io/projected/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-kube-api-access-zvrvq\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321772 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/418b4e01-6bac-4840-8164-88a4f0ef298b-serviceca\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321788 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-systemd-units\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321804 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-node-log\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321819 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-log-socket\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321835 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-cni-bin\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321852 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321870 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-cnibin\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321888 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321914 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-slash\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321941 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovnkube-script-lib\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321967 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-kubernetes\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.321984 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysctl-d\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322002 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-lib-modules\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322019 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-var-lib-kubelet\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.322947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322036 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-cni-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322067 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47rgs\" (UniqueName: \"kubernetes.io/projected/03fb8246-7a76-492f-9e8b-b658d2a5fd49-kube-api-access-47rgs\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322084 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4gwv\" (UniqueName: \"kubernetes.io/projected/8b8b5eb2-9602-4550-9841-d06829964c04-kube-api-access-f4gwv\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322103 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-run-netns\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322130 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovn-node-metrics-cert\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322148 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-sys-fs\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322163 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-tuned\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322200 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-tmp\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322263 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-cni-binary-copy\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322310 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-os-release\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322355 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-netns\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322604 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-cni-multus\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322651 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-multus-certs\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322736 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-etc-kubernetes\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322747 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-daemon-config\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322796 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-system-cni-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322835 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-run-k8s-cni-cncf-io\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322847 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-hostroot\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.323774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.322990 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-host-var-lib-cni-bin\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.324684 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.323552 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-multus-cni-dir\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.328335 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.328310 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:22.328335 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.328337 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:22.328478 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.328354 2557 projected.go:196] Error preparing data for projected volume kube-api-access-dprfj for pod openshift-network-diagnostics/network-check-target-hlh6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:22.328478 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.328435 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj podName:31ecce0b-4f6f-4c06-acd2-fba6be692e14 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:22.828404765 +0000 UTC m=+2.934368917 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-dprfj" (UniqueName: "kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj") pod "network-check-target-hlh6j" (UID: "31ecce0b-4f6f-4c06-acd2-fba6be692e14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:22.329414 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.329387 2557 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 21 21:48:22.333632 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.333610 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvrvq\" (UniqueName: \"kubernetes.io/projected/54532f1d-dd88-45ab-ac92-b0fac6cd4e7a-kube-api-access-zvrvq\") pod \"multus-cgt88\" (UID: \"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a\") " pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.391341 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.391276 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" event={"ID":"06a81050eca56907ec38c411d5d99cf2","Type":"ContainerStarted","Data":"1a4e396189f6dc4e71e6dad4992352a170de3e7fb284720d6a8475c5b8170e24"} Jul 21 21:48:22.394227 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.394190 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" event={"ID":"e5af688db312735faccb6800239fbc06","Type":"ContainerStarted","Data":"80db9cee3e3c12b2e892ee31c6b671f5fb1cabc2a1c2abb7d8b34a15fa0e6fbb"} Jul 21 21:48:22.422572 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422529 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.422734 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422577 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysctl-conf\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.422734 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422605 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-sys\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.422734 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422629 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c22209d7-ce32-4882-870d-1ec2f71fe97a-dbus\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.422734 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422654 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sv6lp\" (UniqueName: \"kubernetes.io/projected/4eef30ff-e679-4d2e-bf2b-121e202bf436-kube-api-access-sv6lp\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:22.422734 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422679 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9c99127-500f-425b-91ce-a6472115be95-host-slash\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.422734 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422706 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e43a69f9-df08-4cfc-9704-a153830b131e-konnectivity-ca\") pod \"konnectivity-agent-rbwsp\" (UID: \"e43a69f9-df08-4cfc-9704-a153830b131e\") " pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:22.422734 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422724 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-cni-netd\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422750 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-registration-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422768 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-etc-selinux\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422783 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9c99127-500f-425b-91ce-a6472115be95-iptables-alerter-script\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422798 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c22209d7-ce32-4882-870d-1ec2f71fe97a-kubelet-config\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422801 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysctl-conf\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422815 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-env-overrides\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422838 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-socket-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422833 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422853 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-run\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422871 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03fb8246-7a76-492f-9e8b-b658d2a5fd49-tmp-dir\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422870 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-cni-netd\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422955 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.422994 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-device-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-host\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423051 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423080 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-etc-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.423181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423107 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-cni-binary-copy\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423172 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovnkube-config\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423197 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-modprobe-d\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423222 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvqvg\" (UniqueName: \"kubernetes.io/projected/418b4e01-6bac-4840-8164-88a4f0ef298b-kube-api-access-xvqvg\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423249 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-kubelet\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423288 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-systemd\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-os-release\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423339 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423366 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-ovn\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423391 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-env-overrides\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423398 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03fb8246-7a76-492f-9e8b-b658d2a5fd49-hosts-file\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e43a69f9-df08-4cfc-9704-a153830b131e-agent-certs\") pod \"konnectivity-agent-rbwsp\" (UID: \"e43a69f9-df08-4cfc-9704-a153830b131e\") " pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423451 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/418b4e01-6bac-4840-8164-88a4f0ef298b-host\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423456 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-registration-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423467 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/d9c99127-500f-425b-91ce-a6472115be95-iptables-alerter-script\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423520 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-var-lib-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424084 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423548 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-system-cni-dir\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423566 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-etc-selinux\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423585 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-socket-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423669 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c22209d7-ce32-4882-870d-1ec2f71fe97a-dbus\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423677 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4k9sv\" (UniqueName: \"kubernetes.io/projected/6907d1d1-462d-4594-ba2f-7c101f2f30fe-kube-api-access-4k9sv\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423707 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pc8bq\" (UniqueName: \"kubernetes.io/projected/2b46dedd-8359-48ea-9a02-784534fcc254-kube-api-access-pc8bq\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423722 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c22209d7-ce32-4882-870d-1ec2f71fe97a-kubelet-config\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423734 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysconfig\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423764 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/418b4e01-6bac-4840-8164-88a4f0ef298b-serviceca\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423791 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-systemd-units\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423797 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d9c99127-500f-425b-91ce-a6472115be95-host-slash\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423826 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-node-log\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423839 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-systemd-units\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423852 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-log-socket\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423882 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-cni-bin\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423922 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423949 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-cnibin\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.424954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424003 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-slash\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424025 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovnkube-config\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovnkube-script-lib\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424082 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-kubernetes\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424110 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysctl-d\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424137 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-lib-modules\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424141 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03fb8246-7a76-492f-9e8b-b658d2a5fd49-tmp-dir\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-var-lib-kubelet\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424224 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.423763 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-system-cni-dir\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424291 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/418b4e01-6bac-4840-8164-88a4f0ef298b-host\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424300 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47rgs\" (UniqueName: \"kubernetes.io/projected/03fb8246-7a76-492f-9e8b-b658d2a5fd49-kube-api-access-47rgs\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424332 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4gwv\" (UniqueName: \"kubernetes.io/projected/8b8b5eb2-9602-4550-9841-d06829964c04-kube-api-access-f4gwv\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424362 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-run-netns\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424389 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovn-node-metrics-cert\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424418 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-sys-fs\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.425838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424453 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-tuned\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424481 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-tmp\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424528 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-systemd\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424575 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-var-lib-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424580 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-systemd\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424702 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovnkube-script-lib\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424825 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-lib-modules\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.424872 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysconfig\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425187 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e43a69f9-df08-4cfc-9704-a153830b131e-konnectivity-ca\") pod \"konnectivity-agent-rbwsp\" (UID: \"e43a69f9-df08-4cfc-9704-a153830b131e\") " pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425225 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-cni-binary-copy\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425250 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-node-log\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425305 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-cni-bin\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425336 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/418b4e01-6bac-4840-8164-88a4f0ef298b-serviceca\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425346 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-slash\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-run\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425400 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-cnibin\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425424 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425456 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.426685 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425492 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-modprobe-d\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425491 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zswj\" (UniqueName: \"kubernetes.io/projected/d9c99127-500f-425b-91ce-a6472115be95-kube-api-access-9zswj\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425507 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/03fb8246-7a76-492f-9e8b-b658d2a5fd49-hosts-file\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425536 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ftl4\" (UniqueName: \"kubernetes.io/projected/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-kube-api-access-4ftl4\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425550 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-kubelet\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425620 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-systemd\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425673 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8b8b5eb2-9602-4550-9841-d06829964c04-os-release\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425718 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-sys\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425732 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-log-socket\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425763 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-var-lib-kubelet\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425778 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425807 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-kubernetes\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425863 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425910 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-sys-fs\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.425948 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-run-ovn\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.426162 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-host\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.426289 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:22.427593 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.426372 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret podName:c22209d7-ce32-4882-870d-1ec2f71fe97a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:22.926350666 +0000 UTC m=+3.032314800 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret") pod "global-pull-secret-syncer-szw9m" (UID: "c22209d7-ce32-4882-870d-1ec2f71fe97a") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.426363 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-sysctl-d\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.426411 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.426412 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2b46dedd-8359-48ea-9a02-784534fcc254-device-dir\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.426444 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-run-netns\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.427002 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs podName:4eef30ff-e679-4d2e-bf2b-121e202bf436 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:22.926981811 +0000 UTC m=+3.032945942 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs") pod "network-metrics-daemon-9fdk4" (UID: "4eef30ff-e679-4d2e-bf2b-121e202bf436") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.427050 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-etc-openvswitch\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.427096 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6907d1d1-462d-4594-ba2f-7c101f2f30fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.427142 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.428531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.427300 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8b8b5eb2-9602-4550-9841-d06829964c04-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.430437 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.430407 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6907d1d1-462d-4594-ba2f-7c101f2f30fe-ovn-node-metrics-cert\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.430555 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.430412 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-etc-tuned\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.431798 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.431752 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e43a69f9-df08-4cfc-9704-a153830b131e-agent-certs\") pod \"konnectivity-agent-rbwsp\" (UID: \"e43a69f9-df08-4cfc-9704-a153830b131e\") " pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:22.432566 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.432543 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-tmp\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.435115 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.435091 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sv6lp\" (UniqueName: \"kubernetes.io/projected/4eef30ff-e679-4d2e-bf2b-121e202bf436-kube-api-access-sv6lp\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:22.442409 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.442349 2557 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 21 21:48:22.442942 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.442914 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47rgs\" (UniqueName: \"kubernetes.io/projected/03fb8246-7a76-492f-9e8b-b658d2a5fd49-kube-api-access-47rgs\") pod \"node-resolver-gptzn\" (UID: \"03fb8246-7a76-492f-9e8b-b658d2a5fd49\") " pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.447200 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.447173 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pc8bq\" (UniqueName: \"kubernetes.io/projected/2b46dedd-8359-48ea-9a02-784534fcc254-kube-api-access-pc8bq\") pod \"aws-ebs-csi-driver-node-lfsj8\" (UID: \"2b46dedd-8359-48ea-9a02-784534fcc254\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.447330 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.447159 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zswj\" (UniqueName: \"kubernetes.io/projected/d9c99127-500f-425b-91ce-a6472115be95-kube-api-access-9zswj\") pod \"iptables-alerter-bqcmq\" (UID: \"d9c99127-500f-425b-91ce-a6472115be95\") " pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.448562 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.448525 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ftl4\" (UniqueName: \"kubernetes.io/projected/aba6184e-bbf6-4b5f-8a2c-859c21456fb4-kube-api-access-4ftl4\") pod \"tuned-nlfmw\" (UID: \"aba6184e-bbf6-4b5f-8a2c-859c21456fb4\") " pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.448562 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.448535 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4gwv\" (UniqueName: \"kubernetes.io/projected/8b8b5eb2-9602-4550-9841-d06829964c04-kube-api-access-f4gwv\") pod \"multus-additional-cni-plugins-9xcsz\" (UID: \"8b8b5eb2-9602-4550-9841-d06829964c04\") " pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.448968 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.448943 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4k9sv\" (UniqueName: \"kubernetes.io/projected/6907d1d1-462d-4594-ba2f-7c101f2f30fe-kube-api-access-4k9sv\") pod \"ovnkube-node-z9qc4\" (UID: \"6907d1d1-462d-4594-ba2f-7c101f2f30fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.450533 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.450503 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvqvg\" (UniqueName: \"kubernetes.io/projected/418b4e01-6bac-4840-8164-88a4f0ef298b-kube-api-access-xvqvg\") pod \"node-ca-cdjcd\" (UID: \"418b4e01-6bac-4840-8164-88a4f0ef298b\") " pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.519945 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.519637 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cgt88" Jul 21 21:48:22.530323 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.529952 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:22.534402 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:22.534372 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54532f1d_dd88_45ab_ac92_b0fac6cd4e7a.slice/crio-c42eb9b4838db5fcdf66648840e8db2590fdca27297f00d4b666ce5d5b8b79a1 WatchSource:0}: Error finding container c42eb9b4838db5fcdf66648840e8db2590fdca27297f00d4b666ce5d5b8b79a1: Status 404 returned error can't find the container with id c42eb9b4838db5fcdf66648840e8db2590fdca27297f00d4b666ce5d5b8b79a1 Jul 21 21:48:22.543114 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:22.542607 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6907d1d1_462d_4594_ba2f_7c101f2f30fe.slice/crio-9f86dbf9206a1466da2d768845bb323491e70f7e809032415a574956c2b57daf WatchSource:0}: Error finding container 9f86dbf9206a1466da2d768845bb323491e70f7e809032415a574956c2b57daf: Status 404 returned error can't find the container with id 9f86dbf9206a1466da2d768845bb323491e70f7e809032415a574956c2b57daf Jul 21 21:48:22.543114 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.542695 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" Jul 21 21:48:22.550134 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.550108 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" Jul 21 21:48:22.553667 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:22.553639 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b46dedd_8359_48ea_9a02_784534fcc254.slice/crio-0ccf19bfa448bbb11c936267b4599e74453609571d9b06a699c4fa42fbb1b2a9 WatchSource:0}: Error finding container 0ccf19bfa448bbb11c936267b4599e74453609571d9b06a699c4fa42fbb1b2a9: Status 404 returned error can't find the container with id 0ccf19bfa448bbb11c936267b4599e74453609571d9b06a699c4fa42fbb1b2a9 Jul 21 21:48:22.558937 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.558911 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-bqcmq" Jul 21 21:48:22.560670 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:22.560641 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba6184e_bbf6_4b5f_8a2c_859c21456fb4.slice/crio-5c8dd637ebffd04a757f7eb6b5802a0416c89e713384f0d06ce6586ae5b7666c WatchSource:0}: Error finding container 5c8dd637ebffd04a757f7eb6b5802a0416c89e713384f0d06ce6586ae5b7666c: Status 404 returned error can't find the container with id 5c8dd637ebffd04a757f7eb6b5802a0416c89e713384f0d06ce6586ae5b7666c Jul 21 21:48:22.567657 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.567636 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gptzn" Jul 21 21:48:22.571672 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:22.571648 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c99127_500f_425b_91ce_a6472115be95.slice/crio-7669fbe45dea3136ce00e8f928d7bce9ec841ad95cc0b0a385f1bd1dd3339e66 WatchSource:0}: Error finding container 7669fbe45dea3136ce00e8f928d7bce9ec841ad95cc0b0a385f1bd1dd3339e66: Status 404 returned error can't find the container with id 7669fbe45dea3136ce00e8f928d7bce9ec841ad95cc0b0a385f1bd1dd3339e66 Jul 21 21:48:22.575979 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.575953 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:22.584445 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.584371 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-cdjcd" Jul 21 21:48:22.588762 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:22.588587 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode43a69f9_df08_4cfc_9704_a153830b131e.slice/crio-04537c79a986a8ea32f025fe5638d0dcc4aa26a9aba2ca6f35b93fcfa799aafe WatchSource:0}: Error finding container 04537c79a986a8ea32f025fe5638d0dcc4aa26a9aba2ca6f35b93fcfa799aafe: Status 404 returned error can't find the container with id 04537c79a986a8ea32f025fe5638d0dcc4aa26a9aba2ca6f35b93fcfa799aafe Jul 21 21:48:22.590953 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.590927 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" Jul 21 21:48:22.602328 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:22.602299 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod418b4e01_6bac_4840_8164_88a4f0ef298b.slice/crio-015bf90f22af08a521eed12f809b20315dc3d4921fc7fbfedbdf954507732bb0 WatchSource:0}: Error finding container 015bf90f22af08a521eed12f809b20315dc3d4921fc7fbfedbdf954507732bb0: Status 404 returned error can't find the container with id 015bf90f22af08a521eed12f809b20315dc3d4921fc7fbfedbdf954507732bb0 Jul 21 21:48:22.607174 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:22.607151 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b8b5eb2_9602_4550_9841_d06829964c04.slice/crio-963d702c1d17e80ca3117a6ab52206d130aafb8a1a74430162833829560509e1 WatchSource:0}: Error finding container 963d702c1d17e80ca3117a6ab52206d130aafb8a1a74430162833829560509e1: Status 404 returned error can't find the container with id 963d702c1d17e80ca3117a6ab52206d130aafb8a1a74430162833829560509e1 Jul 21 21:48:22.829441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.829338 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:22.829621 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.829552 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:22.829621 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.829579 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:22.829621 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.829594 2557 projected.go:196] Error preparing data for projected volume kube-api-access-dprfj for pod openshift-network-diagnostics/network-check-target-hlh6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:22.829782 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.829664 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj podName:31ecce0b-4f6f-4c06-acd2-fba6be692e14 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:23.829643611 +0000 UTC m=+3.935607757 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-dprfj" (UniqueName: "kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj") pod "network-check-target-hlh6j" (UID: "31ecce0b-4f6f-4c06-acd2-fba6be692e14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:22.930837 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.930790 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:22.931020 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:22.930897 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:22.931020 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.930991 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:22.931020 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.931006 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:22.931239 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.931066 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs podName:4eef30ff-e679-4d2e-bf2b-121e202bf436 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:23.931050229 +0000 UTC m=+4.037014356 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs") pod "network-metrics-daemon-9fdk4" (UID: "4eef30ff-e679-4d2e-bf2b-121e202bf436") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:22.931239 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:22.931081 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret podName:c22209d7-ce32-4882-870d-1ec2f71fe97a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:23.931074571 +0000 UTC m=+4.037038696 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret") pod "global-pull-secret-syncer-szw9m" (UID: "c22209d7-ce32-4882-870d-1ec2f71fe97a") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:23.255819 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.255690 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 21:43:23 +0000 UTC" deadline="2028-01-01 23:25:17.160436362 +0000 UTC" Jul 21 21:48:23.255819 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.255774 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12697h36m53.9046758s" Jul 21 21:48:23.398546 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.398429 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bqcmq" event={"ID":"d9c99127-500f-425b-91ce-a6472115be95","Type":"ContainerStarted","Data":"7669fbe45dea3136ce00e8f928d7bce9ec841ad95cc0b0a385f1bd1dd3339e66"} Jul 21 21:48:23.400470 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.400435 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" event={"ID":"aba6184e-bbf6-4b5f-8a2c-859c21456fb4","Type":"ContainerStarted","Data":"5c8dd637ebffd04a757f7eb6b5802a0416c89e713384f0d06ce6586ae5b7666c"} Jul 21 21:48:23.401968 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.401922 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cdjcd" event={"ID":"418b4e01-6bac-4840-8164-88a4f0ef298b","Type":"ContainerStarted","Data":"015bf90f22af08a521eed12f809b20315dc3d4921fc7fbfedbdf954507732bb0"} Jul 21 21:48:23.405299 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.405269 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rbwsp" event={"ID":"e43a69f9-df08-4cfc-9704-a153830b131e","Type":"ContainerStarted","Data":"04537c79a986a8ea32f025fe5638d0dcc4aa26a9aba2ca6f35b93fcfa799aafe"} Jul 21 21:48:23.407119 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.407075 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gptzn" event={"ID":"03fb8246-7a76-492f-9e8b-b658d2a5fd49","Type":"ContainerStarted","Data":"a7c64e3cda134411ac621672bd59d62bc29f841884db2f29c4e53dcbbb53adc7"} Jul 21 21:48:23.408759 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.408735 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" event={"ID":"2b46dedd-8359-48ea-9a02-784534fcc254","Type":"ContainerStarted","Data":"0ccf19bfa448bbb11c936267b4599e74453609571d9b06a699c4fa42fbb1b2a9"} Jul 21 21:48:23.410265 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.410237 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"9f86dbf9206a1466da2d768845bb323491e70f7e809032415a574956c2b57daf"} Jul 21 21:48:23.412563 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.412534 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cgt88" event={"ID":"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a","Type":"ContainerStarted","Data":"c42eb9b4838db5fcdf66648840e8db2590fdca27297f00d4b666ce5d5b8b79a1"} Jul 21 21:48:23.414284 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.414252 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" event={"ID":"8b8b5eb2-9602-4550-9841-d06829964c04","Type":"ContainerStarted","Data":"963d702c1d17e80ca3117a6ab52206d130aafb8a1a74430162833829560509e1"} Jul 21 21:48:23.840332 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.840238 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:23.840520 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:23.840438 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:23.840520 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:23.840460 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:23.840520 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:23.840476 2557 projected.go:196] Error preparing data for projected volume kube-api-access-dprfj for pod openshift-network-diagnostics/network-check-target-hlh6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:23.840692 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:23.840540 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj podName:31ecce0b-4f6f-4c06-acd2-fba6be692e14 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:25.840519359 +0000 UTC m=+5.946483488 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-dprfj" (UniqueName: "kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj") pod "network-check-target-hlh6j" (UID: "31ecce0b-4f6f-4c06-acd2-fba6be692e14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:23.941587 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.941540 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:23.941762 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:23.941612 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:23.941844 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:23.941784 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:23.941898 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:23.941849 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret podName:c22209d7-ce32-4882-870d-1ec2f71fe97a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:25.941829226 +0000 UTC m=+6.047793355 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret") pod "global-pull-secret-syncer-szw9m" (UID: "c22209d7-ce32-4882-870d-1ec2f71fe97a") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:23.942359 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:23.942336 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:23.942462 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:23.942401 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs podName:4eef30ff-e679-4d2e-bf2b-121e202bf436 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:25.942381936 +0000 UTC m=+6.048346064 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs") pod "network-metrics-daemon-9fdk4" (UID: "4eef30ff-e679-4d2e-bf2b-121e202bf436") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:24.256407 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:24.256279 2557 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 21:43:23 +0000 UTC" deadline="2028-04-05 01:29:48.60102607 +0000 UTC" Jul 21 21:48:24.256407 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:24.256347 2557 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14955h41m24.344683424s" Jul 21 21:48:24.387013 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:24.386977 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:24.387206 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:24.387132 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:24.387834 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:24.387543 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:24.387834 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:24.387670 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:24.387834 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:24.387706 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:24.387834 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:24.387785 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:25.433969 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:25.433677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" event={"ID":"06a81050eca56907ec38c411d5d99cf2","Type":"ContainerStarted","Data":"f33bf619e8bed2b12fd468c96d61054ab156ac44bcc5124f27c3ee267349fa4c"} Jul 21 21:48:25.437914 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:25.437877 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" event={"ID":"e5af688db312735faccb6800239fbc06","Type":"ContainerStarted","Data":"55378c806da1d8314e4239c9798954da54c85bbd7b1d14534c95a096279f5942"} Jul 21 21:48:25.860798 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:25.860744 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:25.861029 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:25.860937 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:25.861029 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:25.860964 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:25.861029 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:25.860979 2557 projected.go:196] Error preparing data for projected volume kube-api-access-dprfj for pod openshift-network-diagnostics/network-check-target-hlh6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:25.861220 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:25.861044 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj podName:31ecce0b-4f6f-4c06-acd2-fba6be692e14 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:29.861024366 +0000 UTC m=+9.966988510 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-dprfj" (UniqueName: "kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj") pod "network-check-target-hlh6j" (UID: "31ecce0b-4f6f-4c06-acd2-fba6be692e14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:25.961858 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:25.961811 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:25.962068 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:25.961933 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:25.962138 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:25.962078 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:25.962257 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:25.962180 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs podName:4eef30ff-e679-4d2e-bf2b-121e202bf436 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:29.96215529 +0000 UTC m=+10.068119422 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs") pod "network-metrics-daemon-9fdk4" (UID: "4eef30ff-e679-4d2e-bf2b-121e202bf436") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:25.962632 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:25.962611 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:25.962711 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:25.962674 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret podName:c22209d7-ce32-4882-870d-1ec2f71fe97a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:29.962657527 +0000 UTC m=+10.068621656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret") pod "global-pull-secret-syncer-szw9m" (UID: "c22209d7-ce32-4882-870d-1ec2f71fe97a") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:26.386318 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:26.386281 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:26.386527 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:26.386421 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:26.386598 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:26.386531 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:26.386667 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:26.386641 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:26.386731 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:26.386707 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:26.386824 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:26.386804 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:28.384328 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:28.384235 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:28.384328 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:28.384258 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:28.384820 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:28.384379 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:28.384820 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:28.384476 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:28.384820 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:28.384545 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:28.384820 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:28.384686 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:28.448796 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:28.448757 2557 generic.go:356] "Generic (PLEG): container finished" podID="06a81050eca56907ec38c411d5d99cf2" containerID="f33bf619e8bed2b12fd468c96d61054ab156ac44bcc5124f27c3ee267349fa4c" exitCode=0 Jul 21 21:48:28.448983 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:28.448815 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" event={"ID":"06a81050eca56907ec38c411d5d99cf2","Type":"ContainerDied","Data":"f33bf619e8bed2b12fd468c96d61054ab156ac44bcc5124f27c3ee267349fa4c"} Jul 21 21:48:28.468305 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:28.468235 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-36.ec2.internal" podStartSLOduration=6.46821237 podStartE2EDuration="6.46821237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:25.473404969 +0000 UTC m=+5.579369136" watchObservedRunningTime="2026-07-21 21:48:28.46821237 +0000 UTC m=+8.574176522" Jul 21 21:48:29.897068 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:29.897023 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:29.897578 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:29.897223 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:29.897578 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:29.897241 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:29.897578 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:29.897251 2557 projected.go:196] Error preparing data for projected volume kube-api-access-dprfj for pod openshift-network-diagnostics/network-check-target-hlh6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:29.897578 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:29.897314 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj podName:31ecce0b-4f6f-4c06-acd2-fba6be692e14 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:37.897300066 +0000 UTC m=+18.003264191 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-dprfj" (UniqueName: "kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj") pod "network-check-target-hlh6j" (UID: "31ecce0b-4f6f-4c06-acd2-fba6be692e14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:29.998434 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:29.998379 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:29.998633 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:29.998450 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:29.998705 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:29.998654 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:29.998747 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:29.998727 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret podName:c22209d7-ce32-4882-870d-1ec2f71fe97a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:37.998705098 +0000 UTC m=+18.104669227 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret") pod "global-pull-secret-syncer-szw9m" (UID: "c22209d7-ce32-4882-870d-1ec2f71fe97a") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:29.999255 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:29.999213 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:29.999383 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:29.999283 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs podName:4eef30ff-e679-4d2e-bf2b-121e202bf436 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:37.999261363 +0000 UTC m=+18.105225489 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs") pod "network-metrics-daemon-9fdk4" (UID: "4eef30ff-e679-4d2e-bf2b-121e202bf436") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:30.384920 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:30.384830 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:30.385084 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:30.384951 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:30.385291 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:30.385265 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:30.385425 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:30.385374 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:30.385499 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:30.385475 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:30.385572 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:30.385475 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:32.384155 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:32.384009 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:32.384645 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:32.384169 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:32.384719 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:32.384647 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:32.384792 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:32.384767 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:32.385069 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:32.384942 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:32.385185 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:32.385075 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:34.384293 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:34.384255 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:34.384740 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:34.384269 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:34.384740 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:34.384400 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:34.384740 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:34.384274 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:34.384740 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:34.384468 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:34.384740 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:34.384595 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:36.384462 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:36.384410 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:36.384972 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:36.384416 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:36.384972 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:36.384549 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:36.384972 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:36.384632 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:36.384972 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:36.384700 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:36.384972 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:36.384774 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:37.960293 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:37.960259 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:37.960782 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:37.960399 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:37.960782 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:37.960416 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:37.960782 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:37.960425 2557 projected.go:196] Error preparing data for projected volume kube-api-access-dprfj for pod openshift-network-diagnostics/network-check-target-hlh6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:37.960782 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:37.960476 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj podName:31ecce0b-4f6f-4c06-acd2-fba6be692e14 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:53.960462775 +0000 UTC m=+34.066426898 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-dprfj" (UniqueName: "kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj") pod "network-check-target-hlh6j" (UID: "31ecce0b-4f6f-4c06-acd2-fba6be692e14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:38.061287 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:38.061244 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:38.061447 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:38.061302 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:38.061447 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:38.061407 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:38.061447 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:38.061436 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:38.061565 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:38.061472 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs podName:4eef30ff-e679-4d2e-bf2b-121e202bf436 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:54.061453527 +0000 UTC m=+34.167417653 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs") pod "network-metrics-daemon-9fdk4" (UID: "4eef30ff-e679-4d2e-bf2b-121e202bf436") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:38.061565 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:38.061490 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret podName:c22209d7-ce32-4882-870d-1ec2f71fe97a nodeName:}" failed. No retries permitted until 2026-07-21 21:48:54.06148177 +0000 UTC m=+34.167445896 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret") pod "global-pull-secret-syncer-szw9m" (UID: "c22209d7-ce32-4882-870d-1ec2f71fe97a") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:38.383942 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:38.383863 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:38.384140 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:38.383863 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:38.384140 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:38.383971 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:38.384140 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:38.383867 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:38.384308 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:38.384139 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:38.384308 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:38.384034 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:40.385082 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:40.385045 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:40.385518 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:40.385243 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:40.385518 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:40.385274 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:40.385518 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:40.385345 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:40.385518 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:40.385399 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:40.385518 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:40.385440 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:42.383559 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:42.383521 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:42.383923 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:42.383522 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:42.383923 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:42.383637 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:42.383923 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:42.383522 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:42.383923 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:42.383707 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:42.383923 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:42.383778 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:43.479301 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.479036 2557 generic.go:356] "Generic (PLEG): container finished" podID="8b8b5eb2-9602-4550-9841-d06829964c04" containerID="7c027c0433b2f6614cd8a72bd088da9dbc27d8d4b35e34ac4b738d36f7052739" exitCode=0 Jul 21 21:48:43.480081 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.479124 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" event={"ID":"8b8b5eb2-9602-4550-9841-d06829964c04","Type":"ContainerDied","Data":"7c027c0433b2f6614cd8a72bd088da9dbc27d8d4b35e34ac4b738d36f7052739"} Jul 21 21:48:43.481109 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.481059 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" event={"ID":"aba6184e-bbf6-4b5f-8a2c-859c21456fb4","Type":"ContainerStarted","Data":"e7fb6491d50304d8a414ef72fa02235448869bde7a948520c4a40cab335d337b"} Jul 21 21:48:43.482599 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.482575 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-cdjcd" event={"ID":"418b4e01-6bac-4840-8164-88a4f0ef298b","Type":"ContainerStarted","Data":"9e3c6fe6a184b362fcb5d35ac9400e69c3c4f10cc616e9e3f2f9b9802c0a0a02"} Jul 21 21:48:43.484026 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.483995 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-rbwsp" event={"ID":"e43a69f9-df08-4cfc-9704-a153830b131e","Type":"ContainerStarted","Data":"20e6fd4905de7a3d73407eae88a2b4e1903fc902f8ff0fa5a67ab698f8218e2d"} Jul 21 21:48:43.485522 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.485495 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gptzn" event={"ID":"03fb8246-7a76-492f-9e8b-b658d2a5fd49","Type":"ContainerStarted","Data":"7d9112be5e55e19c46cc8348eb431b4a9d00647c76410604a51bdfc9861a4312"} Jul 21 21:48:43.487414 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.487394 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" event={"ID":"2b46dedd-8359-48ea-9a02-784534fcc254","Type":"ContainerStarted","Data":"34069dc927d0edfa752b1cc1a1545d3a1b8054ad3809353caee67843f1fb6b8f"} Jul 21 21:48:43.490511 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.490486 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"28afc6cb487b1c38eb55da74f28f21b218e8d01a5743157e03333a687838ef8d"} Jul 21 21:48:43.490604 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.490565 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"31a17aa270efb7343d74b3516cbcb164547c25680e2a50bf14c3c56b23061577"} Jul 21 21:48:43.490664 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.490640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"f0ea840501a5a288190f44fadc9181d8c02abc5b7b849b0ea56b4112845203cc"} Jul 21 21:48:43.490707 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.490687 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"2c3b92f51ca8cd499c5028d01a6371fc51686f7c464c91172d6bb70da9c736c4"} Jul 21 21:48:43.490746 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.490737 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"162eb2e2038f6e1ba7f0cc761874f00dac50714ec6e3326bb50f0fa011b36c08"} Jul 21 21:48:43.490789 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.490772 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"63ffc56cbb495680c4adf0d33f32aa908e617e01390c9d711c193253d4506a59"} Jul 21 21:48:43.492246 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.492224 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cgt88" event={"ID":"54532f1d-dd88-45ab-ac92-b0fac6cd4e7a","Type":"ContainerStarted","Data":"e109229c39d31681f9e7bbbfb0cf45444d146f3fe93587d85a283cd590605640"} Jul 21 21:48:43.494503 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.494484 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" event={"ID":"06a81050eca56907ec38c411d5d99cf2","Type":"ContainerStarted","Data":"5b068f90f93780735e47919cee57ba44e3b4414ce92662027a4c3c41e0d50fe0"} Jul 21 21:48:43.523942 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.523885 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-cdjcd" podStartSLOduration=2.564131808 podStartE2EDuration="22.523870994s" totalImagesPullingTime="19.959739186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:43.523613507 +0000 UTC m=+23.629577654" watchObservedRunningTime="2026-07-21 21:48:43.523870994 +0000 UTC m=+23.629835140" Jul 21 21:48:43.538156 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.538083 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-36.ec2.internal" podStartSLOduration=21.538066215 podStartE2EDuration="21.538066215s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:43.537515641 +0000 UTC m=+23.643479788" watchObservedRunningTime="2026-07-21 21:48:43.538066215 +0000 UTC m=+23.644030360" Jul 21 21:48:43.559620 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.559553 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cgt88" podStartSLOduration=2.470678598 podStartE2EDuration="22.559535719s" totalImagesPullingTime="20.088857121s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:43.558945281 +0000 UTC m=+23.664909428" watchObservedRunningTime="2026-07-21 21:48:43.559535719 +0000 UTC m=+23.665499864" Jul 21 21:48:43.576969 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.576916 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-nlfmw" podStartSLOduration=2.567672782 podStartE2EDuration="22.576897184s" totalImagesPullingTime="20.009224402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:43.576548982 +0000 UTC m=+23.682513153" watchObservedRunningTime="2026-07-21 21:48:43.576897184 +0000 UTC m=+23.682861332" Jul 21 21:48:43.599397 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:43.599348 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gptzn" podStartSLOduration=2.840679306 podStartE2EDuration="22.599332661s" totalImagesPullingTime="19.758653355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:43.59874692 +0000 UTC m=+23.704711067" watchObservedRunningTime="2026-07-21 21:48:43.599332661 +0000 UTC m=+23.705296807" Jul 21 21:48:44.132452 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.132422 2557 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 21 21:48:44.279805 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.279633 2557 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T21:48:44.132444997Z","UUID":"a094e5b1-eee9-4c4e-ac3d-ef759d60eb56","Handler":null,"Name":"","Endpoint":""} Jul 21 21:48:44.281585 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.281559 2557 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 21 21:48:44.281742 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.281606 2557 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 21 21:48:44.383935 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.383895 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:44.384156 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.383896 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:44.384156 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:44.384023 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:44.384156 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.383895 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:44.384156 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:44.384127 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:44.384357 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:44.384201 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:44.498973 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.498941 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" event={"ID":"2b46dedd-8359-48ea-9a02-784534fcc254","Type":"ContainerStarted","Data":"808e289512066d9893c997e4e15c8ac3114a5913221a3f08773636683d6c88d5"} Jul 21 21:48:44.501332 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.500912 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-bqcmq" event={"ID":"d9c99127-500f-425b-91ce-a6472115be95","Type":"ContainerStarted","Data":"443c6e4e681977a95b9b6db685d107e4fe4c470c715503b40a20bd13f81561c0"} Jul 21 21:48:44.516558 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.516509 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-rbwsp" podStartSLOduration=3.5466585630000003 podStartE2EDuration="23.516491043s" totalImagesPullingTime="19.96983248s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:43.620053024 +0000 UTC m=+23.726017190" watchObservedRunningTime="2026-07-21 21:48:44.516491043 +0000 UTC m=+24.622455190" Jul 21 21:48:44.516870 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:44.516832 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-bqcmq" podStartSLOduration=7.593161296 podStartE2EDuration="23.516826342s" totalImagesPullingTime="15.923665046s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:44.516414103 +0000 UTC m=+24.622378254" watchObservedRunningTime="2026-07-21 21:48:44.516826342 +0000 UTC m=+24.622790487" Jul 21 21:48:45.506556 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:45.506515 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" event={"ID":"2b46dedd-8359-48ea-9a02-784534fcc254","Type":"ContainerStarted","Data":"44229d7cf7172e2ed0ef3409e2cc28bb133ea71d645a1ca89f78fcce96815307"} Jul 21 21:48:46.384562 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:46.384368 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:46.384882 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:46.384368 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:46.384882 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:46.384661 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:46.384882 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:46.384372 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:46.384882 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:46.384738 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:46.384882 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:46.384836 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:46.512355 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:46.512316 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"5e47bb0a531cba81aed586e2ebc88d56c283241d1e5eb7a5a030bc79b316a512"} Jul 21 21:48:48.080687 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:48.080606 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:48.081257 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:48.081234 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:48.097420 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:48.097376 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lfsj8" podStartSLOduration=4.672026296 podStartE2EDuration="27.097358449s" totalImagesPullingTime="22.425332153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:45.538908052 +0000 UTC m=+25.644872218" watchObservedRunningTime="2026-07-21 21:48:48.097358449 +0000 UTC m=+28.203322594" Jul 21 21:48:48.383987 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:48.383786 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:48.384271 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:48.383786 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:48.384271 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:48.384082 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:48.384271 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:48.384184 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:48.384271 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:48.383818 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:48.384553 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:48.384288 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:48.520689 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:48.520660 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" event={"ID":"6907d1d1-462d-4594-ba2f-7c101f2f30fe","Type":"ContainerStarted","Data":"543d7d60627fe3dc3b846ff5301ced65a21571b06ff973e7f57a7a8a33ca6903"} Jul 21 21:48:48.547862 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:48.547814 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" podStartSLOduration=7.458890637 podStartE2EDuration="27.547797586s" totalImagesPullingTime="20.088906949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:48:48.547305003 +0000 UTC m=+28.653269169" watchObservedRunningTime="2026-07-21 21:48:48.547797586 +0000 UTC m=+28.653761732" Jul 21 21:48:49.525053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:49.525016 2557 generic.go:356] "Generic (PLEG): container finished" podID="8b8b5eb2-9602-4550-9841-d06829964c04" containerID="737c8525c8a51fee860b82435fc43bf6567ee7d6edd17a43b5bc04462e6aa4cc" exitCode=0 Jul 21 21:48:49.525937 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:49.525342 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" event={"ID":"8b8b5eb2-9602-4550-9841-d06829964c04","Type":"ContainerDied","Data":"737c8525c8a51fee860b82435fc43bf6567ee7d6edd17a43b5bc04462e6aa4cc"} Jul 21 21:48:49.525937 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:49.525808 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:49.525937 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:49.525882 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:49.525937 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:49.525928 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:49.549111 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:49.549052 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:49.551895 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:49.551874 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:48:50.281235 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.281196 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hlh6j"] Jul 21 21:48:50.281428 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.281332 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:50.281498 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:50.281423 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:50.284938 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.284912 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9fdk4"] Jul 21 21:48:50.285075 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.284966 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-szw9m"] Jul 21 21:48:50.285075 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.285024 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:50.285210 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:50.285111 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:50.285210 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.285167 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:50.285314 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:50.285237 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:50.409623 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.409593 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:50.409808 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.409714 2557 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness" Jul 21 21:48:50.410175 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:50.410152 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-rbwsp" Jul 21 21:48:51.383540 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:51.383507 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:51.383937 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:51.383615 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:51.531680 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:51.531646 2557 generic.go:356] "Generic (PLEG): container finished" podID="8b8b5eb2-9602-4550-9841-d06829964c04" containerID="c5d6278a26531f53cf12073302c89df3c274895d790cf31027b483be4a5cb987" exitCode=0 Jul 21 21:48:51.531843 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:51.531711 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" event={"ID":"8b8b5eb2-9602-4550-9841-d06829964c04","Type":"ContainerDied","Data":"c5d6278a26531f53cf12073302c89df3c274895d790cf31027b483be4a5cb987"} Jul 21 21:48:52.384226 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:52.384023 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:52.384648 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:52.384023 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:52.384648 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:52.384361 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:52.384648 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:52.384490 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:53.383606 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:53.383570 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:53.383790 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:53.383686 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:53.538592 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:53.538559 2557 generic.go:356] "Generic (PLEG): container finished" podID="8b8b5eb2-9602-4550-9841-d06829964c04" containerID="1e7cf2d1be35475fbc1852e9e205dc6f3da12e0fb721a8753242b6593b8ba21a" exitCode=0 Jul 21 21:48:53.539121 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:53.538601 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" event={"ID":"8b8b5eb2-9602-4550-9841-d06829964c04","Type":"ContainerDied","Data":"1e7cf2d1be35475fbc1852e9e205dc6f3da12e0fb721a8753242b6593b8ba21a"} Jul 21 21:48:53.980080 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:53.980040 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:53.980282 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:53.980213 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 21 21:48:53.980282 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:53.980236 2557 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 21 21:48:53.980282 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:53.980246 2557 projected.go:196] Error preparing data for projected volume kube-api-access-dprfj for pod openshift-network-diagnostics/network-check-target-hlh6j: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:53.980399 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:53.980317 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj podName:31ecce0b-4f6f-4c06-acd2-fba6be692e14 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:25.980301475 +0000 UTC m=+66.086265599 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-dprfj" (UniqueName: "kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj") pod "network-check-target-hlh6j" (UID: "31ecce0b-4f6f-4c06-acd2-fba6be692e14") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 21 21:48:54.081186 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:54.081151 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:54.081186 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:54.081191 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:54.081388 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:54.081324 2557 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:54.081388 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:54.081326 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:54.081388 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:54.081376 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret podName:c22209d7-ce32-4882-870d-1ec2f71fe97a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:26.081363703 +0000 UTC m=+66.187327830 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret") pod "global-pull-secret-syncer-szw9m" (UID: "c22209d7-ce32-4882-870d-1ec2f71fe97a") : object "kube-system"/"original-pull-secret" not registered Jul 21 21:48:54.081388 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:54.081390 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs podName:4eef30ff-e679-4d2e-bf2b-121e202bf436 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:26.08138339 +0000 UTC m=+66.187347514 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs") pod "network-metrics-daemon-9fdk4" (UID: "4eef30ff-e679-4d2e-bf2b-121e202bf436") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 21 21:48:54.383837 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:54.383586 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:54.383837 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:54.383633 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:54.383837 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:54.383733 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hlh6j" podUID="31ecce0b-4f6f-4c06-acd2-fba6be692e14" Jul 21 21:48:54.384148 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:54.383873 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-szw9m" podUID="c22209d7-ce32-4882-870d-1ec2f71fe97a" Jul 21 21:48:55.384103 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:55.384065 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:55.384551 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:55.384240 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9fdk4" podUID="4eef30ff-e679-4d2e-bf2b-121e202bf436" Jul 21 21:48:56.253853 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.253826 2557 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-36.ec2.internal" event="NodeReady" Jul 21 21:48:56.254025 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.253970 2557 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 21 21:48:56.289031 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.288991 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cfc58fd6d-hhtvl"] Jul 21 21:48:56.292684 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.292662 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.295048 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.295029 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 21 21:48:56.295177 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.295083 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 21 21:48:56.295373 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.295356 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9js2h\"" Jul 21 21:48:56.295541 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.295522 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 21 21:48:56.301564 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.301546 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 21 21:48:56.307656 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.307551 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cfc58fd6d-hhtvl"] Jul 21 21:48:56.308067 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.307772 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-ck8nd"] Jul 21 21:48:56.312850 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.312826 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2cczj"] Jul 21 21:48:56.312974 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.312956 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.315545 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.315523 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pp5jq\"" Jul 21 21:48:56.315757 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.315741 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 21 21:48:56.315811 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.315802 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 21 21:48:56.316602 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.316583 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2cczj" Jul 21 21:48:56.319172 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.319149 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 21 21:48:56.320886 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.320525 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7wwbw\"" Jul 21 21:48:56.320886 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.320754 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 21 21:48:56.321155 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.321063 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2cczj"] Jul 21 21:48:56.321248 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.321231 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 21 21:48:56.322249 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.321791 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ck8nd"] Jul 21 21:48:56.383581 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.383485 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:48:56.383581 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.383562 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:48:56.386249 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.386221 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 21 21:48:56.386604 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.386255 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fd2qt\"" Jul 21 21:48:56.386604 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.386329 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:48:56.386604 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.386453 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 21 21:48:56.396805 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396785 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36237ef1-41a3-4fe3-bf11-b2f5712df413-tmp-dir\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.396913 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396816 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.396913 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396838 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt6w6\" (UniqueName: \"kubernetes.io/projected/36237ef1-41a3-4fe3-bf11-b2f5712df413-kube-api-access-lt6w6\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.396913 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396862 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-image-registry-private-configuration\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.396913 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396886 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-bound-sa-token\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.396913 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396910 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2db5177f-c854-4edb-85b1-27c573fc9673-ca-trust-extracted\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.397184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396933 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-installation-pull-secrets\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.397184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396960 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6916be75-1fca-4fbd-8bf1-d210b4416423-cert\") pod \"ingress-canary-2cczj\" (UID: \"6916be75-1fca-4fbd-8bf1-d210b4416423\") " pod="openshift-ingress-canary/ingress-canary-2cczj" Jul 21 21:48:56.397184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.396981 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-trusted-ca\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.397184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.397008 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7th2\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-kube-api-access-h7th2\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.397184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.397038 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vlwbj\" (UniqueName: \"kubernetes.io/projected/6916be75-1fca-4fbd-8bf1-d210b4416423-kube-api-access-vlwbj\") pod \"ingress-canary-2cczj\" (UID: \"6916be75-1fca-4fbd-8bf1-d210b4416423\") " pod="openshift-ingress-canary/ingress-canary-2cczj" Jul 21 21:48:56.397184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.397062 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.397184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.397107 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-registry-certificates\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.397184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.397143 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36237ef1-41a3-4fe3-bf11-b2f5712df413-config-volume\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.450978 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.450937 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rbwsp_e43a69f9-df08-4cfc-9704-a153830b131e/konnectivity-agent/0.log" limit=1024 Jul 21 21:48:56.497732 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.497698 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vlwbj\" (UniqueName: \"kubernetes.io/projected/6916be75-1fca-4fbd-8bf1-d210b4416423-kube-api-access-vlwbj\") pod \"ingress-canary-2cczj\" (UID: \"6916be75-1fca-4fbd-8bf1-d210b4416423\") " pod="openshift-ingress-canary/ingress-canary-2cczj" Jul 21 21:48:56.497732 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.497731 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.497955 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.497750 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-registry-certificates\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.497955 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.497770 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36237ef1-41a3-4fe3-bf11-b2f5712df413-config-volume\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.497955 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:56.497874 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:48:56.497955 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:56.497893 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfc58fd6d-hhtvl: secret "image-registry-tls" not found Jul 21 21:48:56.497955 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:56.497952 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls podName:2db5177f-c854-4edb-85b1-27c573fc9673 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:56.997931524 +0000 UTC m=+37.103895651 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls") pod "image-registry-5cfc58fd6d-hhtvl" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673") : secret "image-registry-tls" not found Jul 21 21:48:56.498258 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498020 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36237ef1-41a3-4fe3-bf11-b2f5712df413-tmp-dir\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.498258 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498052 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.498258 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498078 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt6w6\" (UniqueName: \"kubernetes.io/projected/36237ef1-41a3-4fe3-bf11-b2f5712df413-kube-api-access-lt6w6\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.498258 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498129 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-image-registry-private-configuration\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.498258 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498156 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-bound-sa-token\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.498258 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:56.498182 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:48:56.498258 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:56.498236 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls podName:36237ef1-41a3-4fe3-bf11-b2f5712df413 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:56.998218941 +0000 UTC m=+37.104183068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls") pod "dns-default-ck8nd" (UID: "36237ef1-41a3-4fe3-bf11-b2f5712df413") : secret "dns-default-metrics-tls" not found Jul 21 21:48:56.498258 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498186 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2db5177f-c854-4edb-85b1-27c573fc9673-ca-trust-extracted\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.498639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498276 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-installation-pull-secrets\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.498639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498308 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6916be75-1fca-4fbd-8bf1-d210b4416423-cert\") pod \"ingress-canary-2cczj\" (UID: \"6916be75-1fca-4fbd-8bf1-d210b4416423\") " pod="openshift-ingress-canary/ingress-canary-2cczj" Jul 21 21:48:56.498639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498311 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/36237ef1-41a3-4fe3-bf11-b2f5712df413-tmp-dir\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.498639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498333 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-trusted-ca\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.498639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498361 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7th2\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-kube-api-access-h7th2\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.498639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498414 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-registry-certificates\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.498639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498445 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/36237ef1-41a3-4fe3-bf11-b2f5712df413-config-volume\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.498639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.498488 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2db5177f-c854-4edb-85b1-27c573fc9673-ca-trust-extracted\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.499161 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.499142 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-trusted-ca\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.502400 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.502378 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-installation-pull-secrets\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.502400 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.502392 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-image-registry-private-configuration\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.502525 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.502386 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6916be75-1fca-4fbd-8bf1-d210b4416423-cert\") pod \"ingress-canary-2cczj\" (UID: \"6916be75-1fca-4fbd-8bf1-d210b4416423\") " pod="openshift-ingress-canary/ingress-canary-2cczj" Jul 21 21:48:56.507128 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.507057 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt6w6\" (UniqueName: \"kubernetes.io/projected/36237ef1-41a3-4fe3-bf11-b2f5712df413-kube-api-access-lt6w6\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:56.507686 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.507663 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vlwbj\" (UniqueName: \"kubernetes.io/projected/6916be75-1fca-4fbd-8bf1-d210b4416423-kube-api-access-vlwbj\") pod \"ingress-canary-2cczj\" (UID: \"6916be75-1fca-4fbd-8bf1-d210b4416423\") " pod="openshift-ingress-canary/ingress-canary-2cczj" Jul 21 21:48:56.508662 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.508643 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-bound-sa-token\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.508774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.508756 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7th2\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-kube-api-access-h7th2\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:56.540113 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.540058 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rbwsp_e43a69f9-df08-4cfc-9704-a153830b131e/konnectivity-agent/0.log" limit=1024 Jul 21 21:48:56.631025 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.630993 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2cczj" Jul 21 21:48:56.778702 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:56.778514 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2cczj"] Jul 21 21:48:56.782261 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:48:56.782239 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6916be75_1fca_4fbd_8bf1_d210b4416423.slice/crio-3aab6988b082523e1c325be3930c25d288bb84e1a396aeae55b1426f94620c25 WatchSource:0}: Error finding container 3aab6988b082523e1c325be3930c25d288bb84e1a396aeae55b1426f94620c25: Status 404 returned error can't find the container with id 3aab6988b082523e1c325be3930c25d288bb84e1a396aeae55b1426f94620c25 Jul 21 21:48:57.001390 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:57.001350 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:57.001554 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:57.001415 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:57.001554 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:57.001524 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:48:57.001652 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:57.001592 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls podName:36237ef1-41a3-4fe3-bf11-b2f5712df413 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:58.001574883 +0000 UTC m=+38.107539010 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls") pod "dns-default-ck8nd" (UID: "36237ef1-41a3-4fe3-bf11-b2f5712df413") : secret "dns-default-metrics-tls" not found Jul 21 21:48:57.001652 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:57.001525 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:48:57.001652 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:57.001619 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfc58fd6d-hhtvl: secret "image-registry-tls" not found Jul 21 21:48:57.001749 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:57.001665 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls podName:2db5177f-c854-4edb-85b1-27c573fc9673 nodeName:}" failed. No retries permitted until 2026-07-21 21:48:58.001653353 +0000 UTC m=+38.107617502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls") pod "image-registry-5cfc58fd6d-hhtvl" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673") : secret "image-registry-tls" not found Jul 21 21:48:57.383535 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:57.383462 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:48:57.386831 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:57.386803 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n9lxs\"" Jul 21 21:48:57.387269 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:57.387157 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:48:57.550771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:57.550732 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2cczj" event={"ID":"6916be75-1fca-4fbd-8bf1-d210b4416423","Type":"ContainerStarted","Data":"3aab6988b082523e1c325be3930c25d288bb84e1a396aeae55b1426f94620c25"} Jul 21 21:48:58.008848 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:58.008810 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:48:58.009038 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:58.008876 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:48:58.009038 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:58.009010 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:48:58.009038 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:58.009024 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfc58fd6d-hhtvl: secret "image-registry-tls" not found Jul 21 21:48:58.009245 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:58.009079 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls podName:2db5177f-c854-4edb-85b1-27c573fc9673 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:00.009061144 +0000 UTC m=+40.115025269 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls") pod "image-registry-5cfc58fd6d-hhtvl" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673") : secret "image-registry-tls" not found Jul 21 21:48:58.009618 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:58.009519 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:48:58.009618 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:48:58.009580 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls podName:36237ef1-41a3-4fe3-bf11-b2f5712df413 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:00.009563153 +0000 UTC m=+40.115527279 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls") pod "dns-default-ck8nd" (UID: "36237ef1-41a3-4fe3-bf11-b2f5712df413") : secret "dns-default-metrics-tls" not found Jul 21 21:48:59.978764 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:48:59.978726 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rbwsp_e43a69f9-df08-4cfc-9704-a153830b131e/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:00.026764 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:00.026739 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:49:00.026877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:00.026797 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:49:00.027014 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:00.026995 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:49:00.027115 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:00.027025 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfc58fd6d-hhtvl: secret "image-registry-tls" not found Jul 21 21:49:00.027115 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:00.027067 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls podName:2db5177f-c854-4edb-85b1-27c573fc9673 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:04.027053902 +0000 UTC m=+44.133018025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls") pod "image-registry-5cfc58fd6d-hhtvl" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673") : secret "image-registry-tls" not found Jul 21 21:49:00.027115 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:00.026999 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:49:00.027272 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:00.027152 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls podName:36237ef1-41a3-4fe3-bf11-b2f5712df413 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:04.027138986 +0000 UTC m=+44.133103109 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls") pod "dns-default-ck8nd" (UID: "36237ef1-41a3-4fe3-bf11-b2f5712df413") : secret "dns-default-metrics-tls" not found Jul 21 21:49:00.083473 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:00.083449 2557 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-rbwsp_e43a69f9-df08-4cfc-9704-a153830b131e/konnectivity-agent/0.log" limit=1024 Jul 21 21:49:00.558620 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:00.558580 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2cczj" event={"ID":"6916be75-1fca-4fbd-8bf1-d210b4416423","Type":"ContainerStarted","Data":"faf92e0f226c2fcd806437b624d6efbcf2e8ca221a779d5d67fbec24e0b83a5b"} Jul 21 21:49:00.561298 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:00.561270 2557 generic.go:356] "Generic (PLEG): container finished" podID="8b8b5eb2-9602-4550-9841-d06829964c04" containerID="45f1c7a3144cfa2185bb95b4a457612f239befb129acb8c5cfe996511080bd72" exitCode=0 Jul 21 21:49:00.561436 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:00.561315 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" event={"ID":"8b8b5eb2-9602-4550-9841-d06829964c04","Type":"ContainerDied","Data":"45f1c7a3144cfa2185bb95b4a457612f239befb129acb8c5cfe996511080bd72"} Jul 21 21:49:00.573534 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:00.573497 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2cczj" podStartSLOduration=1.351437592 podStartE2EDuration="4.573485035s" totalImagesPullingTime="3.222047443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:00.572782805 +0000 UTC m=+40.678746953" watchObservedRunningTime="2026-07-21 21:49:00.573485035 +0000 UTC m=+40.679449180" Jul 21 21:49:01.566042 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:01.566007 2557 generic.go:356] "Generic (PLEG): container finished" podID="8b8b5eb2-9602-4550-9841-d06829964c04" containerID="be84dff8a2875a25384c83a7f1f3997da924f9132a230c4ea98f14453dc315e8" exitCode=0 Jul 21 21:49:01.566431 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:01.566110 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" event={"ID":"8b8b5eb2-9602-4550-9841-d06829964c04","Type":"ContainerDied","Data":"be84dff8a2875a25384c83a7f1f3997da924f9132a230c4ea98f14453dc315e8"} Jul 21 21:49:02.571877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:02.571842 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" event={"ID":"8b8b5eb2-9602-4550-9841-d06829964c04","Type":"ContainerStarted","Data":"635296c46cbf36a6e536aed8532a5a455a6f7a567c52daadf08e931fe29fff0e"} Jul 21 21:49:02.595693 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:02.595617 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9xcsz" podStartSLOduration=8.085734637 podStartE2EDuration="41.595595558s" totalImagesPullingTime="33.509860921s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:02.593714054 +0000 UTC m=+42.699678199" watchObservedRunningTime="2026-07-21 21:49:02.595595558 +0000 UTC m=+42.701559705" Jul 21 21:49:04.057418 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.057374 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:49:04.057792 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.057434 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:49:04.057792 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.057533 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:49:04.057792 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.057584 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:49:04.057792 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.057596 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls podName:36237ef1-41a3-4fe3-bf11-b2f5712df413 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.057579398 +0000 UTC m=+52.163543522 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls") pod "dns-default-ck8nd" (UID: "36237ef1-41a3-4fe3-bf11-b2f5712df413") : secret "dns-default-metrics-tls" not found Jul 21 21:49:04.057792 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.057598 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfc58fd6d-hhtvl: secret "image-registry-tls" not found Jul 21 21:49:04.057792 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.057643 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls podName:2db5177f-c854-4edb-85b1-27c573fc9673 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.057630571 +0000 UTC m=+52.163594696 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls") pod "image-registry-5cfc58fd6d-hhtvl" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673") : secret "image-registry-tls" not found Jul 21 21:49:04.455700 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.455623 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4"] Jul 21 21:49:04.487201 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.487166 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4"] Jul 21 21:49:04.487345 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.487288 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4" Jul 21 21:49:04.490321 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.490302 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 21 21:49:04.491269 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.491251 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:49:04.491269 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.491265 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-gqkqx\"" Jul 21 21:49:04.555492 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.555464 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm"] Jul 21 21:49:04.561361 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.561328 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5c56w\" (UniqueName: \"kubernetes.io/projected/9a0ff853-f1f9-44b2-96df-6f97300e61ab-kube-api-access-5c56w\") pod \"volume-data-source-validator-78cf4c6f6-j7cc4\" (UID: \"9a0ff853-f1f9-44b2-96df-6f97300e61ab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4" Jul 21 21:49:04.576792 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.576769 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2"] Jul 21 21:49:04.576915 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.576859 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:04.579483 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.579462 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:49:04.579483 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.579472 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 21 21:49:04.579645 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.579493 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4gq5f\"" Jul 21 21:49:04.579785 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.579763 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 21 21:49:04.590870 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.590851 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-gh7jz"] Jul 21 21:49:04.590975 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.590959 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.593710 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.593692 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 21 21:49:04.594871 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.594851 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 21 21:49:04.594992 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.594873 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 21 21:49:04.594992 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.594922 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ksjph\"" Jul 21 21:49:04.594992 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.594961 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 21 21:49:04.607001 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.606983 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2"] Jul 21 21:49:04.607079 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.607008 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-gh7jz"] Jul 21 21:49:04.607079 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.607020 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm"] Jul 21 21:49:04.607164 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.607084 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.609959 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.609938 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 21 21:49:04.610511 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.610491 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 21 21:49:04.611649 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.611613 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 21 21:49:04.611739 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.611661 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8g5lb\"" Jul 21 21:49:04.611739 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.611663 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:49:04.616565 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.616547 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 21 21:49:04.653184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.653148 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2hqv9"] Jul 21 21:49:04.662587 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.662563 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5c56w\" (UniqueName: \"kubernetes.io/projected/9a0ff853-f1f9-44b2-96df-6f97300e61ab-kube-api-access-5c56w\") pod \"volume-data-source-validator-78cf4c6f6-j7cc4\" (UID: \"9a0ff853-f1f9-44b2-96df-6f97300e61ab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4" Jul 21 21:49:04.662662 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.662596 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:04.662662 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.662618 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzm9m\" (UniqueName: \"kubernetes.io/projected/0ac805b8-88a3-4e5e-b43e-05ab738427d2-kube-api-access-nzm9m\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:04.662662 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.662634 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzwqr\" (UniqueName: \"kubernetes.io/projected/67567445-406c-4492-b0f3-1a484f765c4f-kube-api-access-rzwqr\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.662662 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.662652 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67567445-406c-4492-b0f3-1a484f765c4f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.662805 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.662701 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.675005 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.674983 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2hqv9"] Jul 21 21:49:04.675167 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.675074 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.677223 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.677192 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-8649769886-nbx92"] Jul 21 21:49:04.677600 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.677573 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 21 21:49:04.677830 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.677816 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 21 21:49:04.677910 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.677864 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 21 21:49:04.678557 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.678541 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 21 21:49:04.678826 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.678813 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-ldxxs\"" Jul 21 21:49:04.681184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.681163 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5c56w\" (UniqueName: \"kubernetes.io/projected/9a0ff853-f1f9-44b2-96df-6f97300e61ab-kube-api-access-5c56w\") pod \"volume-data-source-validator-78cf4c6f6-j7cc4\" (UID: \"9a0ff853-f1f9-44b2-96df-6f97300e61ab\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4" Jul 21 21:49:04.684601 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.684582 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 21 21:49:04.693972 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.693956 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.697671 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.697651 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 21 21:49:04.698499 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.698407 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 21 21:49:04.698499 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.698419 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 21 21:49:04.698793 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.698776 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 21 21:49:04.698964 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.698949 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 21 21:49:04.699149 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.699133 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 21 21:49:04.699419 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.699403 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dzr99\"" Jul 21 21:49:04.699912 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.699888 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8649769886-nbx92"] Jul 21 21:49:04.763943 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.763906 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:04.763943 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.763950 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-serving-cert\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.764187 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.763983 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25671a47-f968-4834-8534-1f0e3e92f43b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.764187 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.763999 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25671a47-f968-4834-8534-1f0e3e92f43b-service-ca-bundle\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.764187 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764014 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-trusted-ca\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.764187 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764031 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67567445-406c-4492-b0f3-1a484f765c4f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.764187 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764056 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2qlb\" (UniqueName: \"kubernetes.io/projected/25671a47-f968-4834-8534-1f0e3e92f43b-kube-api-access-z2qlb\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.764187 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.764070 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 21:49:04.764187 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.764162 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls podName:0ac805b8-88a3-4e5e-b43e-05ab738427d2 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:05.264146656 +0000 UTC m=+45.370110779 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m8csm" (UID: "0ac805b8-88a3-4e5e-b43e-05ab738427d2") : secret "samples-operator-tls" not found Jul 21 21:49:04.764441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764221 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzwqr\" (UniqueName: \"kubernetes.io/projected/67567445-406c-4492-b0f3-1a484f765c4f-kube-api-access-rzwqr\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.764441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764239 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-config\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.764441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.764441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764290 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25671a47-f968-4834-8534-1f0e3e92f43b-serving-cert\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.764441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764329 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzm9m\" (UniqueName: \"kubernetes.io/projected/0ac805b8-88a3-4e5e-b43e-05ab738427d2-kube-api-access-nzm9m\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:04.764441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764386 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/25671a47-f968-4834-8534-1f0e3e92f43b-snapshots\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.764441 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.764422 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:04.764441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764432 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25671a47-f968-4834-8534-1f0e3e92f43b-tmp\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.764713 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.764487 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls podName:67567445-406c-4492-b0f3-1a484f765c4f nodeName:}" failed. No retries permitted until 2026-07-21 21:49:05.264472967 +0000 UTC m=+45.370437092 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9qff2" (UID: "67567445-406c-4492-b0f3-1a484f765c4f") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:04.764713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764493 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwp7c\" (UniqueName: \"kubernetes.io/projected/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-kube-api-access-jwp7c\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.764793 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.764775 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/67567445-406c-4492-b0f3-1a484f765c4f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.773618 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.773588 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzm9m\" (UniqueName: \"kubernetes.io/projected/0ac805b8-88a3-4e5e-b43e-05ab738427d2-kube-api-access-nzm9m\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:04.773724 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.773639 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzwqr\" (UniqueName: \"kubernetes.io/projected/67567445-406c-4492-b0f3-1a484f765c4f-kube-api-access-rzwqr\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:04.795425 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.795397 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4" Jul 21 21:49:04.866045 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866018 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2qlb\" (UniqueName: \"kubernetes.io/projected/25671a47-f968-4834-8534-1f0e3e92f43b-kube-api-access-z2qlb\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.866249 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866230 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-stats-auth\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.866371 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866354 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-config\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.866463 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866432 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25671a47-f968-4834-8534-1f0e3e92f43b-serving-cert\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.866522 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866488 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.866576 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866551 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/25671a47-f968-4834-8534-1f0e3e92f43b-snapshots\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.866624 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866599 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5hc\" (UniqueName: \"kubernetes.io/projected/37e21193-957c-4af6-b708-b77445d6229a-kube-api-access-bm5hc\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.866675 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866625 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25671a47-f968-4834-8534-1f0e3e92f43b-tmp\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.866675 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866653 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwp7c\" (UniqueName: \"kubernetes.io/projected/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-kube-api-access-jwp7c\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.866772 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866697 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.866772 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866728 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-serving-cert\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.866867 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866774 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25671a47-f968-4834-8534-1f0e3e92f43b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.866867 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866802 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-default-certificate\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.866867 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25671a47-f968-4834-8534-1f0e3e92f43b-service-ca-bundle\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.866867 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.866858 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-trusted-ca\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.867357 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.867335 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/25671a47-f968-4834-8534-1f0e3e92f43b-tmp\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.867746 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.867720 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/25671a47-f968-4834-8534-1f0e3e92f43b-snapshots\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.868194 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.868169 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25671a47-f968-4834-8534-1f0e3e92f43b-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.868292 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.868229 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/25671a47-f968-4834-8534-1f0e3e92f43b-service-ca-bundle\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.870457 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.870431 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-serving-cert\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.870762 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.870740 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/25671a47-f968-4834-8534-1f0e3e92f43b-serving-cert\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.875350 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.875318 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2qlb\" (UniqueName: \"kubernetes.io/projected/25671a47-f968-4834-8534-1f0e3e92f43b-kube-api-access-z2qlb\") pod \"insights-operator-555d9d7885-2hqv9\" (UID: \"25671a47-f968-4834-8534-1f0e3e92f43b\") " pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:04.876268 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.876244 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwp7c\" (UniqueName: \"kubernetes.io/projected/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-kube-api-access-jwp7c\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.877973 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.877933 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-config\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.878148 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.878128 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc-trusted-ca\") pod \"console-operator-565b7b569d-gh7jz\" (UID: \"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc\") " pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.915704 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.915670 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:04.942836 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.942809 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4"] Jul 21 21:49:04.946177 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:04.946150 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a0ff853_f1f9_44b2_96df_6f97300e61ab.slice/crio-d800a6dbd739230cee2c8e47bf12fcdf0881d48c8cbe78deb461aaf7349e91d8 WatchSource:0}: Error finding container d800a6dbd739230cee2c8e47bf12fcdf0881d48c8cbe78deb461aaf7349e91d8: Status 404 returned error can't find the container with id d800a6dbd739230cee2c8e47bf12fcdf0881d48c8cbe78deb461aaf7349e91d8 Jul 21 21:49:04.967829 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.967795 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.967953 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.967854 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5hc\" (UniqueName: \"kubernetes.io/projected/37e21193-957c-4af6-b708-b77445d6229a-kube-api-access-bm5hc\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.968015 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.967946 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:49:04.968015 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.968013 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:05.467997788 +0000 UTC m=+45.573961915 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : secret "router-metrics-certs-default" not found Jul 21 21:49:04.968119 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.968030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.968119 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.968078 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-default-certificate\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.968201 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:04.968187 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:05.468175602 +0000 UTC m=+45.574139743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : configmap references non-existent config key: service-ca.crt Jul 21 21:49:04.968267 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.968245 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-stats-auth\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.970611 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.970587 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-stats-auth\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.970729 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.970667 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-default-certificate\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.978989 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.978964 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5hc\" (UniqueName: \"kubernetes.io/projected/37e21193-957c-4af6-b708-b77445d6229a-kube-api-access-bm5hc\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:04.988192 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:04.988169 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-2hqv9" Jul 21 21:49:05.112812 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.112776 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-gh7jz"] Jul 21 21:49:05.115670 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:05.115639 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bc3b474_56dc_422b_8cb5_26a5bc9ff5cc.slice/crio-e0d3a2a6ca3476d57b72cfe1da9bb76cdc2453799337c6bbbb491c974f267a7b WatchSource:0}: Error finding container e0d3a2a6ca3476d57b72cfe1da9bb76cdc2453799337c6bbbb491c974f267a7b: Status 404 returned error can't find the container with id e0d3a2a6ca3476d57b72cfe1da9bb76cdc2453799337c6bbbb491c974f267a7b Jul 21 21:49:05.149387 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.149355 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-2hqv9"] Jul 21 21:49:05.152276 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:05.152251 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25671a47_f968_4834_8534_1f0e3e92f43b.slice/crio-fed0d420b0607dbcae4423e255c9d0efc33ff3521d8fcddc0f0df706984c7023 WatchSource:0}: Error finding container fed0d420b0607dbcae4423e255c9d0efc33ff3521d8fcddc0f0df706984c7023: Status 404 returned error can't find the container with id fed0d420b0607dbcae4423e255c9d0efc33ff3521d8fcddc0f0df706984c7023 Jul 21 21:49:05.270782 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.270737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:05.270985 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.270864 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:05.270985 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:05.270893 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:05.270985 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:05.270956 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls podName:67567445-406c-4492-b0f3-1a484f765c4f nodeName:}" failed. No retries permitted until 2026-07-21 21:49:06.270940061 +0000 UTC m=+46.376904185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9qff2" (UID: "67567445-406c-4492-b0f3-1a484f765c4f") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:05.271143 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:05.270996 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 21:49:05.271143 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:05.271060 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls podName:0ac805b8-88a3-4e5e-b43e-05ab738427d2 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:06.271046519 +0000 UTC m=+46.377010642 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m8csm" (UID: "0ac805b8-88a3-4e5e-b43e-05ab738427d2") : secret "samples-operator-tls" not found Jul 21 21:49:05.472634 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.472604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:05.472810 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.472696 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:05.472810 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:05.472772 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:49:05.472916 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:05.472836 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:06.472820115 +0000 UTC m=+46.578784242 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : secret "router-metrics-certs-default" not found Jul 21 21:49:05.472916 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:05.472874 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:06.472852521 +0000 UTC m=+46.578816667 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : configmap references non-existent config key: service-ca.crt Jul 21 21:49:05.580796 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.580755 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2hqv9" event={"ID":"25671a47-f968-4834-8534-1f0e3e92f43b","Type":"ContainerStarted","Data":"fed0d420b0607dbcae4423e255c9d0efc33ff3521d8fcddc0f0df706984c7023"} Jul 21 21:49:05.581988 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.581956 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" event={"ID":"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc","Type":"ContainerStarted","Data":"e0d3a2a6ca3476d57b72cfe1da9bb76cdc2453799337c6bbbb491c974f267a7b"} Jul 21 21:49:05.583282 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:05.583262 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4" event={"ID":"9a0ff853-f1f9-44b2-96df-6f97300e61ab","Type":"ContainerStarted","Data":"d800a6dbd739230cee2c8e47bf12fcdf0881d48c8cbe78deb461aaf7349e91d8"} Jul 21 21:49:06.280106 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.280054 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:06.280617 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:06.280216 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 21:49:06.280617 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:06.280284 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls podName:0ac805b8-88a3-4e5e-b43e-05ab738427d2 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:08.280268679 +0000 UTC m=+48.386232806 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m8csm" (UID: "0ac805b8-88a3-4e5e-b43e-05ab738427d2") : secret "samples-operator-tls" not found Jul 21 21:49:06.280735 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:06.280696 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:06.280788 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:06.280751 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls podName:67567445-406c-4492-b0f3-1a484f765c4f nodeName:}" failed. No retries permitted until 2026-07-21 21:49:08.280735554 +0000 UTC m=+48.386699678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9qff2" (UID: "67567445-406c-4492-b0f3-1a484f765c4f") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:06.281184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.281163 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:06.415539 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.415501 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt"] Jul 21 21:49:06.438354 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.438313 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt"] Jul 21 21:49:06.438522 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.438411 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.444692 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.444660 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:49:06.444859 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.444838 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-5fcb7\"" Jul 21 21:49:06.445137 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.445086 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 21 21:49:06.445270 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.445193 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 21 21:49:06.446330 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.446156 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 21 21:49:06.483466 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.483170 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:06.483466 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.483281 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:06.483466 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:06.483314 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:49:06.483466 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:06.483383 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:08.483362307 +0000 UTC m=+48.589326436 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : secret "router-metrics-certs-default" not found Jul 21 21:49:06.483466 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:06.483434 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:08.483418744 +0000 UTC m=+48.589382867 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : configmap references non-existent config key: service-ca.crt Jul 21 21:49:06.584598 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.584523 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/794afb78-cd2f-4061-8a6b-950b5c00240f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.584759 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.584609 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlzdd\" (UniqueName: \"kubernetes.io/projected/794afb78-cd2f-4061-8a6b-950b5c00240f-kube-api-access-qlzdd\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.584759 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.584687 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/794afb78-cd2f-4061-8a6b-950b5c00240f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.685866 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.685823 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/794afb78-cd2f-4061-8a6b-950b5c00240f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.686128 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.685935 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlzdd\" (UniqueName: \"kubernetes.io/projected/794afb78-cd2f-4061-8a6b-950b5c00240f-kube-api-access-qlzdd\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.686128 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.686013 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/794afb78-cd2f-4061-8a6b-950b5c00240f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.686704 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.686637 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/794afb78-cd2f-4061-8a6b-950b5c00240f-config\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.688364 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.688340 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/794afb78-cd2f-4061-8a6b-950b5c00240f-serving-cert\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.696649 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.696622 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlzdd\" (UniqueName: \"kubernetes.io/projected/794afb78-cd2f-4061-8a6b-950b5c00240f-kube-api-access-qlzdd\") pod \"kube-storage-version-migrator-operator-7658d9b749-smvjt\" (UID: \"794afb78-cd2f-4061-8a6b-950b5c00240f\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:06.751316 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:06.751279 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" Jul 21 21:49:08.301065 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.301024 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:08.301469 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.301130 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:08.301469 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:08.301210 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:08.301469 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:08.301242 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 21:49:08.301469 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:08.301287 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls podName:67567445-406c-4492-b0f3-1a484f765c4f nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.301270207 +0000 UTC m=+52.407234330 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9qff2" (UID: "67567445-406c-4492-b0f3-1a484f765c4f") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:08.301469 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:08.301301 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls podName:0ac805b8-88a3-4e5e-b43e-05ab738427d2 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.301294455 +0000 UTC m=+52.407258579 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m8csm" (UID: "0ac805b8-88a3-4e5e-b43e-05ab738427d2") : secret "samples-operator-tls" not found Jul 21 21:49:08.394853 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.394824 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6"] Jul 21 21:49:08.421341 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.421306 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6"] Jul 21 21:49:08.421529 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.421407 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.424216 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.424194 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 21 21:49:08.424428 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.424413 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 21 21:49:08.425300 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.425277 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:49:08.425406 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.425350 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-bnqf5\"" Jul 21 21:49:08.425406 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.425395 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 21 21:49:08.502917 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.502894 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:08.503036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.503021 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:08.503181 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:08.503159 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:49:08.503282 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:08.503222 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.503197229 +0000 UTC m=+52.609161372 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : configmap references non-existent config key: service-ca.crt Jul 21 21:49:08.503282 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:08.503247 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.503236876 +0000 UTC m=+52.609201003 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : secret "router-metrics-certs-default" not found Jul 21 21:49:08.604433 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.604409 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-serving-cert\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.604604 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.604585 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbz5f\" (UniqueName: \"kubernetes.io/projected/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-kube-api-access-lbz5f\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.604687 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.604645 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-config\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.626540 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.626512 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt"] Jul 21 21:49:08.629678 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:08.629653 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794afb78_cd2f_4061_8a6b_950b5c00240f.slice/crio-895f18d8321665427f19a54447b5458dd043f5cd57130a2e380f03884ca64e57 WatchSource:0}: Error finding container 895f18d8321665427f19a54447b5458dd043f5cd57130a2e380f03884ca64e57: Status 404 returned error can't find the container with id 895f18d8321665427f19a54447b5458dd043f5cd57130a2e380f03884ca64e57 Jul 21 21:49:08.705320 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.705292 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbz5f\" (UniqueName: \"kubernetes.io/projected/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-kube-api-access-lbz5f\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.705480 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.705341 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-config\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.705671 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.705646 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-serving-cert\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.705867 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.705845 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-config\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.707989 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.707964 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-serving-cert\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.717376 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.717352 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbz5f\" (UniqueName: \"kubernetes.io/projected/1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b-kube-api-access-lbz5f\") pod \"service-ca-operator-5bcbc94c78-kq4b6\" (UID: \"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b\") " pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.730410 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.730386 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" Jul 21 21:49:08.862619 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:08.862582 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6"] Jul 21 21:49:08.865961 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:08.865925 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc571c7_3bd1_4c91_b5cd_d1fdb74e3a3b.slice/crio-ec61ccbc6dee67b847cff32aa6d5715e43d92b10755a4a791dc73ceeb46bf00f WatchSource:0}: Error finding container ec61ccbc6dee67b847cff32aa6d5715e43d92b10755a4a791dc73ceeb46bf00f: Status 404 returned error can't find the container with id ec61ccbc6dee67b847cff32aa6d5715e43d92b10755a4a791dc73ceeb46bf00f Jul 21 21:49:09.308259 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.308211 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt"] Jul 21 21:49:09.322282 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.322253 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt"] Jul 21 21:49:09.322464 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.322372 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt" Jul 21 21:49:09.325546 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.325521 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8bvdg\"" Jul 21 21:49:09.412038 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.412006 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsj7w\" (UniqueName: \"kubernetes.io/projected/b6937c99-8c9d-44ff-879a-e4da0f5ff84c-kube-api-access-lsj7w\") pod \"network-check-source-7bc5b6bf85-59mpt\" (UID: \"b6937c99-8c9d-44ff-879a-e4da0f5ff84c\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt" Jul 21 21:49:09.512776 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.512735 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsj7w\" (UniqueName: \"kubernetes.io/projected/b6937c99-8c9d-44ff-879a-e4da0f5ff84c-kube-api-access-lsj7w\") pod \"network-check-source-7bc5b6bf85-59mpt\" (UID: \"b6937c99-8c9d-44ff-879a-e4da0f5ff84c\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt" Jul 21 21:49:09.523660 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.523628 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsj7w\" (UniqueName: \"kubernetes.io/projected/b6937c99-8c9d-44ff-879a-e4da0f5ff84c-kube-api-access-lsj7w\") pod \"network-check-source-7bc5b6bf85-59mpt\" (UID: \"b6937c99-8c9d-44ff-879a-e4da0f5ff84c\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt" Jul 21 21:49:09.596955 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.596863 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" event={"ID":"794afb78-cd2f-4061-8a6b-950b5c00240f","Type":"ContainerStarted","Data":"895f18d8321665427f19a54447b5458dd043f5cd57130a2e380f03884ca64e57"} Jul 21 21:49:09.598979 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.598947 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2hqv9" event={"ID":"25671a47-f968-4834-8534-1f0e3e92f43b","Type":"ContainerStarted","Data":"a2bd3b139b4ac6d3303d3d4f248fc7db4a920ac9707c7e0518ddec3743de41de"} Jul 21 21:49:09.601852 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.601816 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/0.log" Jul 21 21:49:09.601992 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.601860 2557 generic.go:356] "Generic (PLEG): container finished" podID="3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc" containerID="3a78ce0ed8c841ab506cb1c0f48338c60062c3965c271694a04cb5ba6ac1d72b" exitCode=255 Jul 21 21:49:09.601992 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.601937 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" event={"ID":"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc","Type":"ContainerDied","Data":"3a78ce0ed8c841ab506cb1c0f48338c60062c3965c271694a04cb5ba6ac1d72b"} Jul 21 21:49:09.602161 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.602140 2557 scope.go:117] "RemoveContainer" containerID="3a78ce0ed8c841ab506cb1c0f48338c60062c3965c271694a04cb5ba6ac1d72b" Jul 21 21:49:09.605324 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.605297 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4" event={"ID":"9a0ff853-f1f9-44b2-96df-6f97300e61ab","Type":"ContainerStarted","Data":"ee65b4b43da073b1e8964c7823fdf2b2147c0ada49e01cd9e9a793e1d5733b6e"} Jul 21 21:49:09.607533 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.607506 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" event={"ID":"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b","Type":"ContainerStarted","Data":"ec61ccbc6dee67b847cff32aa6d5715e43d92b10755a4a791dc73ceeb46bf00f"} Jul 21 21:49:09.619139 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.619080 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-2hqv9" podStartSLOduration=2.270676372 podStartE2EDuration="5.619067741s" totalImagesPullingTime="3.348391369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:09.617961699 +0000 UTC m=+49.723925844" watchObservedRunningTime="2026-07-21 21:49:09.619067741 +0000 UTC m=+49.725031886" Jul 21 21:49:09.632894 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.632867 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt" Jul 21 21:49:09.633533 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.633482 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-j7cc4" podStartSLOduration=2.089314125 podStartE2EDuration="5.633464215s" totalImagesPullingTime="3.54415009s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:09.632745795 +0000 UTC m=+49.738709963" watchObservedRunningTime="2026-07-21 21:49:09.633464215 +0000 UTC m=+49.739428362" Jul 21 21:49:09.852978 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:09.852888 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt"] Jul 21 21:49:09.889253 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:09.889201 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6937c99_8c9d_44ff_879a_e4da0f5ff84c.slice/crio-a741648af16a9421b822f6ecd642cc49611ec0a1296be2bb2c50a7ec86e036f7 WatchSource:0}: Error finding container a741648af16a9421b822f6ecd642cc49611ec0a1296be2bb2c50a7ec86e036f7: Status 404 returned error can't find the container with id a741648af16a9421b822f6ecd642cc49611ec0a1296be2bb2c50a7ec86e036f7 Jul 21 21:49:10.252352 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.252319 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-zbgfp"] Jul 21 21:49:10.263144 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.263118 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.266772 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.266175 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 21 21:49:10.266772 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.266234 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zbgfp"] Jul 21 21:49:10.266772 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.266494 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tqm4\"" Jul 21 21:49:10.266772 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.266540 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 21 21:49:10.422920 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.422882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.423405 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.422942 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhv8c\" (UniqueName: \"kubernetes.io/projected/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-api-access-jhv8c\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.423405 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.422982 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d1a845f-de5b-4748-866c-87b10b7ee243-data-volume\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.423405 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.423029 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.423405 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.423171 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d1a845f-de5b-4748-866c-87b10b7ee243-crio-socket\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.524707 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.524614 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhv8c\" (UniqueName: \"kubernetes.io/projected/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-api-access-jhv8c\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.524707 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.524678 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d1a845f-de5b-4748-866c-87b10b7ee243-data-volume\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.524926 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.524737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.524926 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.524804 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d1a845f-de5b-4748-866c-87b10b7ee243-crio-socket\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.524926 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.524883 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.525083 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:10.525003 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:10.525083 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:10.525047 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:49:10.525083 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:10.525058 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls podName:3d1a845f-de5b-4748-866c-87b10b7ee243 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:11.025042946 +0000 UTC m=+51.131007070 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbgfp" (UID: "3d1a845f-de5b-4748-866c-87b10b7ee243") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:10.525291 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:10.525123 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm podName:3d1a845f-de5b-4748-866c-87b10b7ee243 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:11.025104701 +0000 UTC m=+51.131068838 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbgfp" (UID: "3d1a845f-de5b-4748-866c-87b10b7ee243") : configmap "kube-rbac-proxy" not found Jul 21 21:49:10.525438 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.525418 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/3d1a845f-de5b-4748-866c-87b10b7ee243-crio-socket\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.525495 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.525419 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/3d1a845f-de5b-4748-866c-87b10b7ee243-data-volume\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.535485 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.535461 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhv8c\" (UniqueName: \"kubernetes.io/projected/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-api-access-jhv8c\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:10.612058 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.612027 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 21:49:10.612543 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.612521 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/0.log" Jul 21 21:49:10.612642 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.612563 2557 generic.go:356] "Generic (PLEG): container finished" podID="3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc" containerID="9ec658f66d2c69f09f396483705bc90dfb405f93466d3ddfcfd25c23ef2ad5d1" exitCode=255 Jul 21 21:49:10.612693 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.612656 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" event={"ID":"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc","Type":"ContainerDied","Data":"9ec658f66d2c69f09f396483705bc90dfb405f93466d3ddfcfd25c23ef2ad5d1"} Jul 21 21:49:10.612735 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.612722 2557 scope.go:117] "RemoveContainer" containerID="3a78ce0ed8c841ab506cb1c0f48338c60062c3965c271694a04cb5ba6ac1d72b" Jul 21 21:49:10.613056 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.613028 2557 scope.go:117] "RemoveContainer" containerID="9ec658f66d2c69f09f396483705bc90dfb405f93466d3ddfcfd25c23ef2ad5d1" Jul 21 21:49:10.613287 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:10.613261 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-gh7jz_openshift-console-operator(3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc)\"" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" podUID="3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc" Jul 21 21:49:10.614366 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:10.614307 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt" event={"ID":"b6937c99-8c9d-44ff-879a-e4da0f5ff84c","Type":"ContainerStarted","Data":"a741648af16a9421b822f6ecd642cc49611ec0a1296be2bb2c50a7ec86e036f7"} Jul 21 21:49:11.029882 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:11.029841 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:11.030106 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:11.029964 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:11.030106 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:11.030019 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:11.030106 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:11.030063 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:49:11.030106 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:11.030086 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls podName:3d1a845f-de5b-4748-866c-87b10b7ee243 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.030068271 +0000 UTC m=+52.136032395 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbgfp" (UID: "3d1a845f-de5b-4748-866c-87b10b7ee243") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:11.030319 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:11.030133 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm podName:3d1a845f-de5b-4748-866c-87b10b7ee243 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:12.030117453 +0000 UTC m=+52.136081590 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbgfp" (UID: "3d1a845f-de5b-4748-866c-87b10b7ee243") : configmap "kube-rbac-proxy" not found Jul 21 21:49:11.617596 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:11.617550 2557 scope.go:117] "RemoveContainer" containerID="9ec658f66d2c69f09f396483705bc90dfb405f93466d3ddfcfd25c23ef2ad5d1" Jul 21 21:49:11.618047 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:11.617783 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-gh7jz_openshift-console-operator(3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc)\"" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" podUID="3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc" Jul 21 21:49:11.651827 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:11.651798 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gptzn_03fb8246-7a76-492f-9e8b-b658d2a5fd49/dns-node-resolver/0.log" Jul 21 21:49:12.042836 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.042602 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:12.042836 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.042682 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:12.042836 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.042758 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:12.042836 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.042786 2557 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 21 21:49:12.042836 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.042820 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls podName:3d1a845f-de5b-4748-866c-87b10b7ee243 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:14.042803998 +0000 UTC m=+54.148768123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbgfp" (UID: "3d1a845f-de5b-4748-866c-87b10b7ee243") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:12.042836 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.042835 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm podName:3d1a845f-de5b-4748-866c-87b10b7ee243 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:14.042827926 +0000 UTC m=+54.148792050 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm") pod "insights-runtime-extractor-zbgfp" (UID: "3d1a845f-de5b-4748-866c-87b10b7ee243") : configmap "kube-rbac-proxy" not found Jul 21 21:49:12.143594 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.143513 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:49:12.143737 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.143620 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:49:12.143737 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.143670 2557 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 21 21:49:12.143826 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.143752 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls podName:36237ef1-41a3-4fe3-bf11-b2f5712df413 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:28.143731718 +0000 UTC m=+68.249695846 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls") pod "dns-default-ck8nd" (UID: "36237ef1-41a3-4fe3-bf11-b2f5712df413") : secret "dns-default-metrics-tls" not found Jul 21 21:49:12.143826 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.143767 2557 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 21 21:49:12.143826 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.143781 2557 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5cfc58fd6d-hhtvl: secret "image-registry-tls" not found Jul 21 21:49:12.143940 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.143832 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls podName:2db5177f-c854-4edb-85b1-27c573fc9673 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:28.143815967 +0000 UTC m=+68.249780092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls") pod "image-registry-5cfc58fd6d-hhtvl" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673") : secret "image-registry-tls" not found Jul 21 21:49:12.345049 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.345012 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:12.345329 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.345179 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:12.345329 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.345206 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 21:49:12.345329 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.345288 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls podName:0ac805b8-88a3-4e5e-b43e-05ab738427d2 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:20.34526804 +0000 UTC m=+60.451232177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m8csm" (UID: "0ac805b8-88a3-4e5e-b43e-05ab738427d2") : secret "samples-operator-tls" not found Jul 21 21:49:12.345329 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.345318 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:12.345543 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.345373 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls podName:67567445-406c-4492-b0f3-1a484f765c4f nodeName:}" failed. No retries permitted until 2026-07-21 21:49:20.345360993 +0000 UTC m=+60.451325117 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9qff2" (UID: "67567445-406c-4492-b0f3-1a484f765c4f") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:12.452489 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.452328 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cdjcd_418b4e01-6bac-4840-8164-88a4f0ef298b/node-ca/0.log" Jul 21 21:49:12.547585 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.547541 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:12.547836 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.547815 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:12.547919 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.547824 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:49:12.547995 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.547982 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:20.547960818 +0000 UTC m=+60.653924941 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : secret "router-metrics-certs-default" not found Jul 21 21:49:12.548049 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:12.548008 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:20.547997425 +0000 UTC m=+60.653961555 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : configmap references non-existent config key: service-ca.crt Jul 21 21:49:12.623025 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.622930 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" event={"ID":"794afb78-cd2f-4061-8a6b-950b5c00240f","Type":"ContainerStarted","Data":"7249bc45ff43806e3420ea1e87064981958347ccc627edda571074e37065c923"} Jul 21 21:49:12.624774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.624746 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 21:49:12.626560 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.626520 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" event={"ID":"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b","Type":"ContainerStarted","Data":"cb080bc420aa31b6e4cc7bdb903bbb08e777cd57046a63012f2a8474168606f7"} Jul 21 21:49:12.641992 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.641944 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" podStartSLOduration=3.408631842 podStartE2EDuration="6.641927699s" totalImagesPullingTime="3.233295857s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:12.640785326 +0000 UTC m=+52.746749485" watchObservedRunningTime="2026-07-21 21:49:12.641927699 +0000 UTC m=+52.747891845" Jul 21 21:49:12.658679 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:12.658627 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" podStartSLOduration=1.6519971610000002 podStartE2EDuration="4.658609287s" totalImagesPullingTime="3.006612126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:12.657360898 +0000 UTC m=+52.763325059" watchObservedRunningTime="2026-07-21 21:49:12.658609287 +0000 UTC m=+52.764573433" Jul 21 21:49:13.059413 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:13.059301 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2cczj_6916be75-1fca-4fbd-8bf1-d210b4416423/serve-healthcheck-canary/0.log" Jul 21 21:49:14.062393 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:14.062348 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:14.062859 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:14.062459 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:14.062859 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:14.062582 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:14.062859 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:14.062634 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls podName:3d1a845f-de5b-4748-866c-87b10b7ee243 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:18.062621045 +0000 UTC m=+58.168585169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbgfp" (UID: "3d1a845f-de5b-4748-866c-87b10b7ee243") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:14.062859 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:14.062844 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/3d1a845f-de5b-4748-866c-87b10b7ee243-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:14.634767 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:14.634676 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt" event={"ID":"b6937c99-8c9d-44ff-879a-e4da0f5ff84c","Type":"ContainerStarted","Data":"0640609f1316010299c9b2eb5c3dd368926577bcbc7e8a1fe26915550d599ec1"} Jul 21 21:49:14.651269 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:14.651221 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-59mpt" podStartSLOduration=1.269298296 podStartE2EDuration="5.651207572s" totalImagesPullingTime="4.381909276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:14.650295775 +0000 UTC m=+54.756259920" watchObservedRunningTime="2026-07-21 21:49:14.651207572 +0000 UTC m=+54.757171730" Jul 21 21:49:14.916463 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:14.916375 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:14.916463 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:14.916431 2557 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:14.916789 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:14.916774 2557 scope.go:117] "RemoveContainer" containerID="9ec658f66d2c69f09f396483705bc90dfb405f93466d3ddfcfd25c23ef2ad5d1" Jul 21 21:49:14.916942 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:14.916925 2557 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-gh7jz_openshift-console-operator(3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc)\"" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" podUID="3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc" Jul 21 21:49:18.098247 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:18.098208 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:18.098683 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:18.098363 2557 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 21 21:49:18.098683 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:18.098430 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls podName:3d1a845f-de5b-4748-866c-87b10b7ee243 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:26.098413694 +0000 UTC m=+66.204377819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls") pod "insights-runtime-extractor-zbgfp" (UID: "3d1a845f-de5b-4748-866c-87b10b7ee243") : secret "insights-runtime-extractor-tls" not found Jul 21 21:49:20.422241 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:20.422203 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:20.422697 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:20.422313 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:20.422697 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:20.422365 2557 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:20.422697 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:20.422436 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls podName:67567445-406c-4492-b0f3-1a484f765c4f nodeName:}" failed. No retries permitted until 2026-07-21 21:49:36.422416141 +0000 UTC m=+76.528380282 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-9qff2" (UID: "67567445-406c-4492-b0f3-1a484f765c4f") : secret "cluster-monitoring-operator-tls" not found Jul 21 21:49:20.422697 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:20.422487 2557 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 21 21:49:20.422697 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:20.422558 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls podName:0ac805b8-88a3-4e5e-b43e-05ab738427d2 nodeName:}" failed. No retries permitted until 2026-07-21 21:49:36.422540563 +0000 UTC m=+76.528504704 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls") pod "cluster-samples-operator-79b7c55f87-m8csm" (UID: "0ac805b8-88a3-4e5e-b43e-05ab738427d2") : secret "samples-operator-tls" not found Jul 21 21:49:20.623867 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:20.623825 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:20.624028 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:20.623913 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:20.624028 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:20.623968 2557 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 21 21:49:20.624116 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:20.624034 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:36.624019302 +0000 UTC m=+76.729983426 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : secret "router-metrics-certs-default" not found Jul 21 21:49:20.624116 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:20.624067 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle podName:37e21193-957c-4af6-b708-b77445d6229a nodeName:}" failed. No retries permitted until 2026-07-21 21:49:36.624052166 +0000 UTC m=+76.730016293 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle") pod "router-default-8649769886-nbx92" (UID: "37e21193-957c-4af6-b708-b77445d6229a") : configmap references non-existent config key: service-ca.crt Jul 21 21:49:21.551170 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:21.551140 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-z9qc4" Jul 21 21:49:26.074455 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.074417 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:49:26.077272 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.077240 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dprfj\" (UniqueName: \"kubernetes.io/projected/31ecce0b-4f6f-4c06-acd2-fba6be692e14-kube-api-access-dprfj\") pod \"network-check-target-hlh6j\" (UID: \"31ecce0b-4f6f-4c06-acd2-fba6be692e14\") " pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:49:26.095552 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.095519 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-fd2qt\"" Jul 21 21:49:26.103720 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.103691 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:49:26.174936 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.174903 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:49:26.175171 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.174953 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:49:26.175171 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.175002 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:26.177868 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.177841 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 21 21:49:26.177868 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.177867 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 21 21:49:26.178038 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.177918 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/3d1a845f-de5b-4748-866c-87b10b7ee243-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-zbgfp\" (UID: \"3d1a845f-de5b-4748-866c-87b10b7ee243\") " pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:26.185265 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:26.185232 2557 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found Jul 21 21:49:26.185375 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:49:26.185326 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs podName:4eef30ff-e679-4d2e-bf2b-121e202bf436 nodeName:}" failed. No retries permitted until 2026-07-21 21:50:30.185298092 +0000 UTC m=+130.291262230 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs") pod "network-metrics-daemon-9fdk4" (UID: "4eef30ff-e679-4d2e-bf2b-121e202bf436") : secret "metrics-daemon-secret" not found Jul 21 21:49:26.187771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.187739 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c22209d7-ce32-4882-870d-1ec2f71fe97a-original-pull-secret\") pod \"global-pull-secret-syncer-szw9m\" (UID: \"c22209d7-ce32-4882-870d-1ec2f71fe97a\") " pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:49:26.232366 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.232335 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hlh6j"] Jul 21 21:49:26.235587 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:26.235559 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ecce0b_4f6f_4c06_acd2_fba6be692e14.slice/crio-8e72acf151ae45bc34390b88e17ca7ea8a638abac0becdb435361e49241357b1 WatchSource:0}: Error finding container 8e72acf151ae45bc34390b88e17ca7ea8a638abac0becdb435361e49241357b1: Status 404 returned error can't find the container with id 8e72acf151ae45bc34390b88e17ca7ea8a638abac0becdb435361e49241357b1 Jul 21 21:49:26.397028 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.396929 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-szw9m" Jul 21 21:49:26.479495 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.479464 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6tqm4\"" Jul 21 21:49:26.486627 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.486602 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-zbgfp" Jul 21 21:49:26.526661 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.526627 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-szw9m"] Jul 21 21:49:26.529732 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:26.529680 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc22209d7_ce32_4882_870d_1ec2f71fe97a.slice/crio-97806e86d4f070b5c04b047fbf2730a8faae18a3db68412aeecc55b198edce6d WatchSource:0}: Error finding container 97806e86d4f070b5c04b047fbf2730a8faae18a3db68412aeecc55b198edce6d: Status 404 returned error can't find the container with id 97806e86d4f070b5c04b047fbf2730a8faae18a3db68412aeecc55b198edce6d Jul 21 21:49:26.624352 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.624320 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-zbgfp"] Jul 21 21:49:26.627133 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:26.627106 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d1a845f_de5b_4748_866c_87b10b7ee243.slice/crio-65041fd12aa5b1d39894c17cfc763428456198b74788e7286b9e918ce86466d3 WatchSource:0}: Error finding container 65041fd12aa5b1d39894c17cfc763428456198b74788e7286b9e918ce86466d3: Status 404 returned error can't find the container with id 65041fd12aa5b1d39894c17cfc763428456198b74788e7286b9e918ce86466d3 Jul 21 21:49:26.669747 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.669663 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-szw9m" event={"ID":"c22209d7-ce32-4882-870d-1ec2f71fe97a","Type":"ContainerStarted","Data":"97806e86d4f070b5c04b047fbf2730a8faae18a3db68412aeecc55b198edce6d"} Jul 21 21:49:26.670821 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.670792 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbgfp" event={"ID":"3d1a845f-de5b-4748-866c-87b10b7ee243","Type":"ContainerStarted","Data":"65041fd12aa5b1d39894c17cfc763428456198b74788e7286b9e918ce86466d3"} Jul 21 21:49:26.672172 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.672153 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hlh6j" event={"ID":"31ecce0b-4f6f-4c06-acd2-fba6be692e14","Type":"ContainerStarted","Data":"0272917bed25244b7a681b9d0757f253fb4e8d427cae0364e0c6c6e01f4f0c97"} Jul 21 21:49:26.672265 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.672184 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hlh6j" event={"ID":"31ecce0b-4f6f-4c06-acd2-fba6be692e14","Type":"ContainerStarted","Data":"8e72acf151ae45bc34390b88e17ca7ea8a638abac0becdb435361e49241357b1"} Jul 21 21:49:26.672307 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.672293 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:49:26.690029 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:26.689978 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hlh6j" podStartSLOduration=65.689962864 podStartE2EDuration="1m5.689962864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:26.688206329 +0000 UTC m=+66.794170475" watchObservedRunningTime="2026-07-21 21:49:26.689962864 +0000 UTC m=+66.795927007" Jul 21 21:49:28.195822 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.195726 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:49:28.196310 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.195823 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:49:28.199475 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.199446 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"image-registry-5cfc58fd6d-hhtvl\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:49:28.210566 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.210536 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/36237ef1-41a3-4fe3-bf11-b2f5712df413-metrics-tls\") pod \"dns-default-ck8nd\" (UID: \"36237ef1-41a3-4fe3-bf11-b2f5712df413\") " pod="openshift-dns/dns-default-ck8nd" Jul 21 21:49:28.383982 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.383953 2557 scope.go:117] "RemoveContainer" containerID="9ec658f66d2c69f09f396483705bc90dfb405f93466d3ddfcfd25c23ef2ad5d1" Jul 21 21:49:28.407637 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.407607 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-9js2h\"" Jul 21 21:49:28.416235 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.415939 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:49:28.429866 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.429700 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-pp5jq\"" Jul 21 21:49:28.437059 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.437028 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-ck8nd" Jul 21 21:49:28.624496 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.624421 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cfc58fd6d-hhtvl"] Jul 21 21:49:28.627555 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:28.627525 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2db5177f_c854_4edb_85b1_27c573fc9673.slice/crio-3b3c7d8a51cad23a8ef252373272b016f710fdda74262724a7d680af90da7415 WatchSource:0}: Error finding container 3b3c7d8a51cad23a8ef252373272b016f710fdda74262724a7d680af90da7415: Status 404 returned error can't find the container with id 3b3c7d8a51cad23a8ef252373272b016f710fdda74262724a7d680af90da7415 Jul 21 21:49:28.649017 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.648988 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-ck8nd"] Jul 21 21:49:28.653750 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:28.653719 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36237ef1_41a3_4fe3_bf11_b2f5712df413.slice/crio-581c56e8150c3e46c5fca8929dbe18fc6f7ead9c452e8dd30b3dec82fd3a4856 WatchSource:0}: Error finding container 581c56e8150c3e46c5fca8929dbe18fc6f7ead9c452e8dd30b3dec82fd3a4856: Status 404 returned error can't find the container with id 581c56e8150c3e46c5fca8929dbe18fc6f7ead9c452e8dd30b3dec82fd3a4856 Jul 21 21:49:28.681307 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.681237 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" event={"ID":"2db5177f-c854-4edb-85b1-27c573fc9673","Type":"ContainerStarted","Data":"3b3c7d8a51cad23a8ef252373272b016f710fdda74262724a7d680af90da7415"} Jul 21 21:49:28.684042 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.684020 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 21:49:28.684189 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.684144 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" event={"ID":"3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc","Type":"ContainerStarted","Data":"90ce94162dfb62f79caf45f20f9ec3dc31f9951ef6dd1b65c11e7503811c4c06"} Jul 21 21:49:28.684867 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.684682 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:28.685974 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.685941 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ck8nd" event={"ID":"36237ef1-41a3-4fe3-bf11-b2f5712df413","Type":"ContainerStarted","Data":"581c56e8150c3e46c5fca8929dbe18fc6f7ead9c452e8dd30b3dec82fd3a4856"} Jul 21 21:49:28.704601 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:28.704493 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" podStartSLOduration=21.326216241 podStartE2EDuration="24.704470863s" totalImagesPullingTime="3.378254622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:28.703864771 +0000 UTC m=+68.809828942" watchObservedRunningTime="2026-07-21 21:49:28.704470863 +0000 UTC m=+68.810435009" Jul 21 21:49:29.401898 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:29.401859 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-gh7jz" Jul 21 21:49:29.693895 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:29.693816 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" event={"ID":"2db5177f-c854-4edb-85b1-27c573fc9673","Type":"ContainerStarted","Data":"add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc"} Jul 21 21:49:29.693895 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:29.693884 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:49:29.715441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:29.715386 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" podStartSLOduration=68.71536787 podStartE2EDuration="1m8.71536787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:29.71440915 +0000 UTC m=+69.820373295" watchObservedRunningTime="2026-07-21 21:49:29.71536787 +0000 UTC m=+69.821332037" Jul 21 21:49:34.711843 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:34.711806 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbgfp" event={"ID":"3d1a845f-de5b-4748-866c-87b10b7ee243","Type":"ContainerStarted","Data":"3468b2d1f65a1b8cbdf9d06bf3b9b3d3a403a9770d992fe5283186486dd22b56"} Jul 21 21:49:34.717705 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:34.717670 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ck8nd" event={"ID":"36237ef1-41a3-4fe3-bf11-b2f5712df413","Type":"ContainerStarted","Data":"f9113f593f677b5eaa7023210a6dd6058c5a3fe12c4958da8326e9614f2b4730"} Jul 21 21:49:34.717881 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:34.717757 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-ck8nd" event={"ID":"36237ef1-41a3-4fe3-bf11-b2f5712df413","Type":"ContainerStarted","Data":"995844a843a7630f8abd47de419900c5d91f918d6a0c6bebd2dc1e5133f38540"} Jul 21 21:49:34.717975 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:34.717954 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-ck8nd" Jul 21 21:49:34.719749 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:34.719709 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-szw9m" event={"ID":"c22209d7-ce32-4882-870d-1ec2f71fe97a","Type":"ContainerStarted","Data":"9556fe690112745e0efd8259af3ad2af49a4b5e5b33f5b2d9384456784b98bae"} Jul 21 21:49:34.738275 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:34.738206 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-ck8nd" podStartSLOduration=33.32005631 podStartE2EDuration="38.738186703s" totalImagesPullingTime="5.418130393s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:34.73806259 +0000 UTC m=+74.844026765" watchObservedRunningTime="2026-07-21 21:49:34.738186703 +0000 UTC m=+74.844150849" Jul 21 21:49:36.472859 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.472806 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:36.473373 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.472925 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:36.475968 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.475930 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/67567445-406c-4492-b0f3-1a484f765c4f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-9qff2\" (UID: \"67567445-406c-4492-b0f3-1a484f765c4f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:36.476112 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.476063 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/0ac805b8-88a3-4e5e-b43e-05ab738427d2-samples-operator-tls\") pod \"cluster-samples-operator-79b7c55f87-m8csm\" (UID: \"0ac805b8-88a3-4e5e-b43e-05ab738427d2\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:36.674890 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.674840 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:36.675120 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.674954 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:36.675638 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.675599 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/37e21193-957c-4af6-b708-b77445d6229a-service-ca-bundle\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:36.677720 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.677694 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/37e21193-957c-4af6-b708-b77445d6229a-metrics-certs\") pod \"router-default-8649769886-nbx92\" (UID: \"37e21193-957c-4af6-b708-b77445d6229a\") " pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:36.688311 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.688285 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-4gq5f\"" Jul 21 21:49:36.695741 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.695711 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" Jul 21 21:49:36.701476 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.701445 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-ksjph\"" Jul 21 21:49:36.709543 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.709513 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" Jul 21 21:49:36.808643 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.808612 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-dzr99\"" Jul 21 21:49:36.816709 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:36.816685 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:37.716445 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.716393 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-szw9m" podStartSLOduration=69.141374924 podStartE2EDuration="1m16.716376402s" totalImagesPullingTime="7.575001478s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:34.756014169 +0000 UTC m=+74.861978331" watchObservedRunningTime="2026-07-21 21:49:37.716376402 +0000 UTC m=+77.822340542" Jul 21 21:49:37.717646 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.717624 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm"] Jul 21 21:49:37.724924 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.724860 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2"] Jul 21 21:49:37.734209 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.734181 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbgfp" event={"ID":"3d1a845f-de5b-4748-866c-87b10b7ee243","Type":"ContainerStarted","Data":"2c24f510d1737f514fa5db1ffe88b8f99884b094918f306bc1eae33071eadbfa"} Jul 21 21:49:37.737309 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.737262 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" event={"ID":"67567445-406c-4492-b0f3-1a484f765c4f","Type":"ContainerStarted","Data":"03e54b01bc16b51c971599d6a5e0baf8f3321a934d6dd0973f35b4fcc4910e52"} Jul 21 21:49:37.739343 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.739308 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-8649769886-nbx92"] Jul 21 21:49:37.751815 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:37.751775 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37e21193_957c_4af6_b708_b77445d6229a.slice/crio-900e307c594b21dc73172a64eb31395090ee8779475841ba30377475542b0ce6 WatchSource:0}: Error finding container 900e307c594b21dc73172a64eb31395090ee8779475841ba30377475542b0ce6: Status 404 returned error can't find the container with id 900e307c594b21dc73172a64eb31395090ee8779475841ba30377475542b0ce6 Jul 21 21:49:37.835646 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.835575 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-q78cf"] Jul 21 21:49:37.838931 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.838907 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-q78cf" Jul 21 21:49:37.842706 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.842684 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 21 21:49:37.842919 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.842894 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-fglh8\"" Jul 21 21:49:37.843249 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.843234 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 21 21:49:37.858392 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.858362 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-q78cf"] Jul 21 21:49:37.987985 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:37.987954 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8qkx\" (UniqueName: \"kubernetes.io/projected/3d5455ad-df25-46af-9e32-c725ed4267c7-kube-api-access-n8qkx\") pod \"downloads-5bc67dcd46-q78cf\" (UID: \"3d5455ad-df25-46af-9e32-c725ed4267c7\") " pod="openshift-console/downloads-5bc67dcd46-q78cf" Jul 21 21:49:38.088554 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.088472 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8qkx\" (UniqueName: \"kubernetes.io/projected/3d5455ad-df25-46af-9e32-c725ed4267c7-kube-api-access-n8qkx\") pod \"downloads-5bc67dcd46-q78cf\" (UID: \"3d5455ad-df25-46af-9e32-c725ed4267c7\") " pod="openshift-console/downloads-5bc67dcd46-q78cf" Jul 21 21:49:38.102042 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.101977 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8qkx\" (UniqueName: \"kubernetes.io/projected/3d5455ad-df25-46af-9e32-c725ed4267c7-kube-api-access-n8qkx\") pod \"downloads-5bc67dcd46-q78cf\" (UID: \"3d5455ad-df25-46af-9e32-c725ed4267c7\") " pod="openshift-console/downloads-5bc67dcd46-q78cf" Jul 21 21:49:38.148507 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.148467 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-q78cf" Jul 21 21:49:38.152488 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.152462 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-84f95887c8-npd92"] Jul 21 21:49:38.155711 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.155691 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.158353 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.158318 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 21 21:49:38.158468 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.158319 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 21 21:49:38.158468 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.158339 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 21 21:49:38.158833 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.158816 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 21 21:49:38.158931 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.158834 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 21 21:49:38.158931 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.158901 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-2fvdp\"" Jul 21 21:49:38.166648 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.166588 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f95887c8-npd92"] Jul 21 21:49:38.290807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.290766 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82ffw\" (UniqueName: \"kubernetes.io/projected/577524c4-7e26-4dd1-9cdb-0161c2690c34-kube-api-access-82ffw\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.290991 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.290828 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-config\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.290991 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.290864 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-serving-cert\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.290991 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.290922 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-oauth-serving-cert\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.290991 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.290955 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-oauth-config\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.291251 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.291011 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-service-ca\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.310123 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.309929 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-q78cf"] Jul 21 21:49:38.392213 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.392015 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82ffw\" (UniqueName: \"kubernetes.io/projected/577524c4-7e26-4dd1-9cdb-0161c2690c34-kube-api-access-82ffw\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.394459 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.392459 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-config\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.394459 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.392502 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-serving-cert\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.394459 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.392532 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-oauth-serving-cert\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.394459 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.392563 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-oauth-config\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.394459 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.392617 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-service-ca\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.394459 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.393459 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-service-ca\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.394459 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.394077 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-oauth-serving-cert\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.394853 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.394487 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-config\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.396475 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.396446 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-serving-cert\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.397733 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.397709 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-oauth-config\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.405523 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.405482 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-82ffw\" (UniqueName: \"kubernetes.io/projected/577524c4-7e26-4dd1-9cdb-0161c2690c34-kube-api-access-82ffw\") pod \"console-84f95887c8-npd92\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.467312 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.467272 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:38.639788 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.639704 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-84f95887c8-npd92"] Jul 21 21:49:38.645681 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:49:38.645604 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577524c4_7e26_4dd1_9cdb_0161c2690c34.slice/crio-641cd5f16fbca49db2dd3c92a7b92f65639c43c02731045ecd09443785045059 WatchSource:0}: Error finding container 641cd5f16fbca49db2dd3c92a7b92f65639c43c02731045ecd09443785045059: Status 404 returned error can't find the container with id 641cd5f16fbca49db2dd3c92a7b92f65639c43c02731045ecd09443785045059 Jul 21 21:49:38.744335 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.744262 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f95887c8-npd92" event={"ID":"577524c4-7e26-4dd1-9cdb-0161c2690c34","Type":"ContainerStarted","Data":"641cd5f16fbca49db2dd3c92a7b92f65639c43c02731045ecd09443785045059"} Jul 21 21:49:38.746665 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.746629 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-q78cf" event={"ID":"3d5455ad-df25-46af-9e32-c725ed4267c7","Type":"ContainerStarted","Data":"1e738e10b1ceca0fd847526d565a794e80b64758c3b767fa63e78a3ad6e7f3eb"} Jul 21 21:49:38.748342 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.748310 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" event={"ID":"0ac805b8-88a3-4e5e-b43e-05ab738427d2","Type":"ContainerStarted","Data":"51de39d53d4402fae1108dc75a784974a874357112a11d7dd124c78d075e91c1"} Jul 21 21:49:38.751083 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.751053 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8649769886-nbx92" event={"ID":"37e21193-957c-4af6-b708-b77445d6229a","Type":"ContainerStarted","Data":"5dd5d68ef8fb000bc30b01504e3918389614dbd90ec5bc9856a91bfbaa5e13f5"} Jul 21 21:49:38.751212 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.751126 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-8649769886-nbx92" event={"ID":"37e21193-957c-4af6-b708-b77445d6229a","Type":"ContainerStarted","Data":"900e307c594b21dc73172a64eb31395090ee8779475841ba30377475542b0ce6"} Jul 21 21:49:38.780780 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.780715 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-8649769886-nbx92" podStartSLOduration=34.780693576 podStartE2EDuration="34.780693576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:38.779590597 +0000 UTC m=+78.885554792" watchObservedRunningTime="2026-07-21 21:49:38.780693576 +0000 UTC m=+78.886657722" Jul 21 21:49:38.817472 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.817432 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:38.820658 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:38.820632 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:39.760326 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:39.760238 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:39.761547 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:39.761521 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-8649769886-nbx92" Jul 21 21:49:44.725523 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:44.725496 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-ck8nd" Jul 21 21:49:45.789523 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:45.789482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-zbgfp" event={"ID":"3d1a845f-de5b-4748-866c-87b10b7ee243","Type":"ContainerStarted","Data":"82be6dae7fcad51cd9ec2b3530045e8ea30c7d699b01f2c9be379c821ecc84b8"} Jul 21 21:49:45.791009 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:45.790979 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" event={"ID":"67567445-406c-4492-b0f3-1a484f765c4f","Type":"ContainerStarted","Data":"f6f25f66b2a3c6806003a1ac396433611c7a2fe1eb078c80ed8edb5eff94c067"} Jul 21 21:49:45.792438 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:45.792414 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f95887c8-npd92" event={"ID":"577524c4-7e26-4dd1-9cdb-0161c2690c34","Type":"ContainerStarted","Data":"918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35"} Jul 21 21:49:45.794396 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:45.794368 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" event={"ID":"0ac805b8-88a3-4e5e-b43e-05ab738427d2","Type":"ContainerStarted","Data":"8ca45d2187f1f6d569550e2b60388d08576adc7e16650b39e77fe356bd009f47"} Jul 21 21:49:45.794520 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:45.794416 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" event={"ID":"0ac805b8-88a3-4e5e-b43e-05ab738427d2","Type":"ContainerStarted","Data":"60307f549aae1ceaf62eb7955e6aced264cb3fc39da10a57d689b60794a2662a"} Jul 21 21:49:45.831171 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:45.831116 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-9qff2" podStartSLOduration=34.860306163 podStartE2EDuration="41.831084668s" totalImagesPullingTime="6.970778505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:45.830531549 +0000 UTC m=+85.936495710" watchObservedRunningTime="2026-07-21 21:49:45.831084668 +0000 UTC m=+85.937048811" Jul 21 21:49:45.831343 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:45.831282 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-zbgfp" podStartSLOduration=17.905990182 podStartE2EDuration="35.831268985s" totalImagesPullingTime="17.925278803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 21:49:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:45.809527662 +0000 UTC m=+85.915491836" watchObservedRunningTime="2026-07-21 21:49:45.831268985 +0000 UTC m=+85.937233133" Jul 21 21:49:45.855882 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:45.855814 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-84f95887c8-npd92" podStartSLOduration=1.56183894 podStartE2EDuration="7.855793606s" totalImagesPullingTime="6.293954666s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:45.855455732 +0000 UTC m=+85.961419906" watchObservedRunningTime="2026-07-21 21:49:45.855793606 +0000 UTC m=+85.961757783" Jul 21 21:49:48.421169 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:48.421121 2557 patch_prober.go:28] interesting pod/image-registry-5cfc58fd6d-hhtvl container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 21 21:49:48.421578 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:48.421194 2557 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" podUID="2db5177f-c854-4edb-85b1-27c573fc9673" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 21:49:48.468251 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:48.468204 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:48.468447 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:48.468280 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:48.473071 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:48.473048 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:48.491642 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:48.491589 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-79b7c55f87-m8csm" podStartSLOduration=37.581639723 podStartE2EDuration="44.491570659s" totalImagesPullingTime="6.909930936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:45.873114161 +0000 UTC m=+85.979078320" watchObservedRunningTime="2026-07-21 21:49:48.491570659 +0000 UTC m=+88.597534806" Jul 21 21:49:48.811737 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:48.811698 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:49:51.707483 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:51.706754 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:49:56.258239 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:56.258200 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f95887c8-npd92"] Jul 21 21:49:57.679688 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:57.679606 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hlh6j" Jul 21 21:49:57.846306 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:57.846264 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-q78cf" event={"ID":"3d5455ad-df25-46af-9e32-c725ed4267c7","Type":"ContainerStarted","Data":"d21ce69520bd6f0584c9578f59faa17398e690cf400495d7e49f7be43e033a43"} Jul 21 21:49:57.846531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:57.846507 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-q78cf" Jul 21 21:49:57.861854 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:57.861824 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-q78cf" Jul 21 21:49:57.868824 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:57.868774 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-q78cf" podStartSLOduration=1.797330649 podStartE2EDuration="20.868753651s" totalImagesPullingTime="19.071423002s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:49:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:49:57.866735545 +0000 UTC m=+97.972699747" watchObservedRunningTime="2026-07-21 21:49:57.868753651 +0000 UTC m=+97.974717799" Jul 21 21:49:59.777497 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.777459 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-s5lft"] Jul 21 21:49:59.810627 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.810591 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.814550 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.814006 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 21 21:49:59.814550 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.814062 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7blhv\"" Jul 21 21:49:59.814550 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.814062 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 21 21:49:59.814550 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.814217 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 21 21:49:59.814550 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.814407 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 21 21:49:59.991361 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991321 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-tls\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.991552 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991370 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-wtmp\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.991552 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991400 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.991552 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991425 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-root\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.991552 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-textfile\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.991740 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991541 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-sys\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.991740 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991592 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61108f6f-64e9-4d13-936c-2e57d1607da0-metrics-client-ca\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.991740 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991637 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:49:59.991740 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:49:59.991682 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4snf\" (UniqueName: \"kubernetes.io/projected/61108f6f-64e9-4d13-936c-2e57d1607da0-kube-api-access-b4snf\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092270 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092180 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-tls\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092270 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092236 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-wtmp\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092270 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092267 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092526 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092299 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-root\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092526 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092356 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-textfile\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092526 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-sys\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092526 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092392 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-wtmp\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092526 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092407 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61108f6f-64e9-4d13-936c-2e57d1607da0-metrics-client-ca\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092526 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092470 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.092526 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.092514 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4snf\" (UniqueName: \"kubernetes.io/projected/61108f6f-64e9-4d13-936c-2e57d1607da0-kube-api-access-b4snf\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.093116 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.093067 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-root\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.093257 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.093113 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-textfile\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.093257 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.093150 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61108f6f-64e9-4d13-936c-2e57d1607da0-sys\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.093444 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.093425 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-accelerators-collector-config\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.093633 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.093616 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61108f6f-64e9-4d13-936c-2e57d1607da0-metrics-client-ca\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.095343 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.095320 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.095756 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.095734 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61108f6f-64e9-4d13-936c-2e57d1607da0-node-exporter-tls\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.102176 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.102154 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4snf\" (UniqueName: \"kubernetes.io/projected/61108f6f-64e9-4d13-936c-2e57d1607da0-kube-api-access-b4snf\") pod \"node-exporter-s5lft\" (UID: \"61108f6f-64e9-4d13-936c-2e57d1607da0\") " pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.123059 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.123028 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-s5lft" Jul 21 21:50:00.159025 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:50:00.158991 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61108f6f_64e9_4d13_936c_2e57d1607da0.slice/crio-db18d8e3b2de44024ddfb951f137e89edd99c5704ca1a73a48c43452eb33debf WatchSource:0}: Error finding container db18d8e3b2de44024ddfb951f137e89edd99c5704ca1a73a48c43452eb33debf: Status 404 returned error can't find the container with id db18d8e3b2de44024ddfb951f137e89edd99c5704ca1a73a48c43452eb33debf Jul 21 21:50:00.861141 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:00.861077 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5lft" event={"ID":"61108f6f-64e9-4d13-936c-2e57d1607da0","Type":"ContainerStarted","Data":"db18d8e3b2de44024ddfb951f137e89edd99c5704ca1a73a48c43452eb33debf"} Jul 21 21:50:01.089339 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:01.089294 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cfc58fd6d-hhtvl"] Jul 21 21:50:01.867360 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:01.867268 2557 generic.go:356] "Generic (PLEG): container finished" podID="61108f6f-64e9-4d13-936c-2e57d1607da0" containerID="fa42a35e87df178b06552f588c22eff57dc503a4fc436d0e9846a2e37efd3f81" exitCode=0 Jul 21 21:50:01.867815 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:01.867371 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5lft" event={"ID":"61108f6f-64e9-4d13-936c-2e57d1607da0","Type":"ContainerDied","Data":"fa42a35e87df178b06552f588c22eff57dc503a4fc436d0e9846a2e37efd3f81"} Jul 21 21:50:02.874326 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:02.874288 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5lft" event={"ID":"61108f6f-64e9-4d13-936c-2e57d1607da0","Type":"ContainerStarted","Data":"f7e71bb36ab2e08adb99d94cb8901f297207139d999ad7d7fc10500eb2c1785d"} Jul 21 21:50:02.874930 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:02.874351 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-s5lft" event={"ID":"61108f6f-64e9-4d13-936c-2e57d1607da0","Type":"ContainerStarted","Data":"c4d9863b576f4ec7b1db049c204c02136dec8f8bbffbe8ab7caba9624d14d5cc"} Jul 21 21:50:02.896934 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:02.896847 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-s5lft" podStartSLOduration=2.365972763 podStartE2EDuration="3.896829389s" totalImagesPullingTime="1.429327314s" totalInitContainerRuntime="101.529312ms" isStatefulPod=true podCreationTimestamp="2026-07-21 21:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:02.896259376 +0000 UTC m=+103.002223541" watchObservedRunningTime="2026-07-21 21:50:02.896829389 +0000 UTC m=+103.002793535" Jul 21 21:50:04.537359 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.537316 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-746fcd7868-lw76b"] Jul 21 21:50:04.572612 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.572573 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746fcd7868-lw76b"] Jul 21 21:50:04.572794 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.572744 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.581157 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.580896 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 21 21:50:04.637052 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.637010 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-service-ca\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.637256 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.637087 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-serving-cert\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.637256 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.637143 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-oauth-config\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.637256 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.637190 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-oauth-serving-cert\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.637441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.637259 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-console-config\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.637441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.637294 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69nj\" (UniqueName: \"kubernetes.io/projected/cc921385-8886-4469-b9af-134121cb928d-kube-api-access-k69nj\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.637441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.637345 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-trusted-ca-bundle\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.738705 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.738661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-trusted-ca-bundle\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.738907 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.738745 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-service-ca\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.738907 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.738812 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-serving-cert\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.738907 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.738846 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-oauth-config\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.738907 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.738875 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-oauth-serving-cert\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.738907 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.738906 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-console-config\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.739279 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.738928 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69nj\" (UniqueName: \"kubernetes.io/projected/cc921385-8886-4469-b9af-134121cb928d-kube-api-access-k69nj\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.739602 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.739573 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-service-ca\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.739753 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.739725 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-trusted-ca-bundle\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.739998 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.739976 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-console-config\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.740469 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.740449 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-oauth-serving-cert\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.742105 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.742068 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-oauth-config\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.742248 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.742228 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-serving-cert\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.748998 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.748968 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69nj\" (UniqueName: \"kubernetes.io/projected/cc921385-8886-4469-b9af-134121cb928d-kube-api-access-k69nj\") pod \"console-746fcd7868-lw76b\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:04.884051 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:04.883964 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:05.076622 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:05.076449 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-746fcd7868-lw76b"] Jul 21 21:50:05.079627 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:50:05.079587 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc921385_8886_4469_b9af_134121cb928d.slice/crio-c4b876b3ce9b4b20ee73a53bb3cb1c9ad5f8b6bf2fdf9af16dd078cec969086e WatchSource:0}: Error finding container c4b876b3ce9b4b20ee73a53bb3cb1c9ad5f8b6bf2fdf9af16dd078cec969086e: Status 404 returned error can't find the container with id c4b876b3ce9b4b20ee73a53bb3cb1c9ad5f8b6bf2fdf9af16dd078cec969086e Jul 21 21:50:05.888982 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:05.888946 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746fcd7868-lw76b" event={"ID":"cc921385-8886-4469-b9af-134121cb928d","Type":"ContainerStarted","Data":"9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f"} Jul 21 21:50:05.889511 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:05.888998 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746fcd7868-lw76b" event={"ID":"cc921385-8886-4469-b9af-134121cb928d","Type":"ContainerStarted","Data":"c4b876b3ce9b4b20ee73a53bb3cb1c9ad5f8b6bf2fdf9af16dd078cec969086e"} Jul 21 21:50:05.909453 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:05.907999 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-746fcd7868-lw76b" podStartSLOduration=1.907978373 podStartE2EDuration="1.907978373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:50:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:05.907337767 +0000 UTC m=+106.013301950" watchObservedRunningTime="2026-07-21 21:50:05.907978373 +0000 UTC m=+106.013942522" Jul 21 21:50:10.466789 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.466751 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746fcd7868-lw76b"] Jul 21 21:50:10.501508 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.501469 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7996885d64-2kpnv"] Jul 21 21:50:10.523270 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.523226 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7996885d64-2kpnv"] Jul 21 21:50:10.523478 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.523361 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.591732 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.591693 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-oauth-config\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.591918 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.591737 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-service-ca\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.591918 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.591793 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-serving-cert\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.591918 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.591808 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-oauth-serving-cert\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.591918 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.591829 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6mqm\" (UniqueName: \"kubernetes.io/projected/339566fd-1984-405a-9d13-8f9b5cf0bd50-kube-api-access-q6mqm\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.592052 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.591932 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-trusted-ca-bundle\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.592052 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.592017 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-config\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.693205 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.693153 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-oauth-config\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.693205 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.693200 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-service-ca\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.693462 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.693230 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-serving-cert\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.693462 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.693255 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-oauth-serving-cert\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.693462 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.693280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6mqm\" (UniqueName: \"kubernetes.io/projected/339566fd-1984-405a-9d13-8f9b5cf0bd50-kube-api-access-q6mqm\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.693462 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.693329 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-trusted-ca-bundle\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.693462 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.693412 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-config\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.694062 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.694033 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-service-ca\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.694189 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.694134 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-oauth-serving-cert\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.694244 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.694198 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-config\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.694634 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.694609 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-trusted-ca-bundle\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.695900 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.695868 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-oauth-config\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.695983 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.695951 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-serving-cert\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.701866 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.701843 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6mqm\" (UniqueName: \"kubernetes.io/projected/339566fd-1984-405a-9d13-8f9b5cf0bd50-kube-api-access-q6mqm\") pod \"console-7996885d64-2kpnv\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:10.835900 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:10.835858 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:11.006814 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:11.006779 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7996885d64-2kpnv"] Jul 21 21:50:11.009784 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:50:11.009754 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod339566fd_1984_405a_9d13_8f9b5cf0bd50.slice/crio-f12d796ea1aa7c44f973fa550d6fed594f9f6a5cd9391a24f9ff55ef872a6125 WatchSource:0}: Error finding container f12d796ea1aa7c44f973fa550d6fed594f9f6a5cd9391a24f9ff55ef872a6125: Status 404 returned error can't find the container with id f12d796ea1aa7c44f973fa550d6fed594f9f6a5cd9391a24f9ff55ef872a6125 Jul 21 21:50:11.912552 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:11.912509 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7996885d64-2kpnv" event={"ID":"339566fd-1984-405a-9d13-8f9b5cf0bd50","Type":"ContainerStarted","Data":"af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae"} Jul 21 21:50:11.912965 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:11.912561 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7996885d64-2kpnv" event={"ID":"339566fd-1984-405a-9d13-8f9b5cf0bd50","Type":"ContainerStarted","Data":"f12d796ea1aa7c44f973fa550d6fed594f9f6a5cd9391a24f9ff55ef872a6125"} Jul 21 21:50:11.932289 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:11.932240 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7996885d64-2kpnv" podStartSLOduration=1.9322218009999999 podStartE2EDuration="1.932221801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:11.930987033 +0000 UTC m=+112.036951178" watchObservedRunningTime="2026-07-21 21:50:11.932221801 +0000 UTC m=+112.038185947" Jul 21 21:50:14.884586 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:14.884540 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:20.836196 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:20.836151 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:20.836674 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:20.836218 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:20.840958 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:20.840936 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:20.948061 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:20.948027 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:50:21.284677 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.284630 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-84f95887c8-npd92" podUID="577524c4-7e26-4dd1-9cdb-0161c2690c34" containerName="console" containerID="cri-o://918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35" gracePeriod=15 Jul 21 21:50:21.561439 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.561370 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:50:21.695557 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.695515 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-config\") pod \"577524c4-7e26-4dd1-9cdb-0161c2690c34\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " Jul 21 21:50:21.695557 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.695565 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-oauth-config\") pod \"577524c4-7e26-4dd1-9cdb-0161c2690c34\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " Jul 21 21:50:21.695820 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.695598 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-oauth-serving-cert\") pod \"577524c4-7e26-4dd1-9cdb-0161c2690c34\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " Jul 21 21:50:21.695820 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.695622 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-82ffw\" (UniqueName: \"kubernetes.io/projected/577524c4-7e26-4dd1-9cdb-0161c2690c34-kube-api-access-82ffw\") pod \"577524c4-7e26-4dd1-9cdb-0161c2690c34\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " Jul 21 21:50:21.695820 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.695671 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-serving-cert\") pod \"577524c4-7e26-4dd1-9cdb-0161c2690c34\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " Jul 21 21:50:21.695820 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.695708 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-service-ca\") pod \"577524c4-7e26-4dd1-9cdb-0161c2690c34\" (UID: \"577524c4-7e26-4dd1-9cdb-0161c2690c34\") " Jul 21 21:50:21.696146 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.696011 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-config" (OuterVolumeSpecName: "console-config") pod "577524c4-7e26-4dd1-9cdb-0161c2690c34" (UID: "577524c4-7e26-4dd1-9cdb-0161c2690c34"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:21.696146 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.696121 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "577524c4-7e26-4dd1-9cdb-0161c2690c34" (UID: "577524c4-7e26-4dd1-9cdb-0161c2690c34"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:21.696262 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.696237 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-service-ca" (OuterVolumeSpecName: "service-ca") pod "577524c4-7e26-4dd1-9cdb-0161c2690c34" (UID: "577524c4-7e26-4dd1-9cdb-0161c2690c34"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:21.698185 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.698162 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577524c4-7e26-4dd1-9cdb-0161c2690c34-kube-api-access-82ffw" (OuterVolumeSpecName: "kube-api-access-82ffw") pod "577524c4-7e26-4dd1-9cdb-0161c2690c34" (UID: "577524c4-7e26-4dd1-9cdb-0161c2690c34"). InnerVolumeSpecName "kube-api-access-82ffw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:21.698185 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.698166 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "577524c4-7e26-4dd1-9cdb-0161c2690c34" (UID: "577524c4-7e26-4dd1-9cdb-0161c2690c34"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:21.698299 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.698234 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "577524c4-7e26-4dd1-9cdb-0161c2690c34" (UID: "577524c4-7e26-4dd1-9cdb-0161c2690c34"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:21.797065 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.797026 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-oauth-config\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:21.797065 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.797057 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-oauth-serving-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:21.797065 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.797067 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82ffw\" (UniqueName: \"kubernetes.io/projected/577524c4-7e26-4dd1-9cdb-0161c2690c34-kube-api-access-82ffw\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:21.797324 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.797078 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-serving-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:21.797324 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.797120 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-service-ca\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:21.797324 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.797131 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577524c4-7e26-4dd1-9cdb-0161c2690c34-console-config\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:21.950046 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.949962 2557 generic.go:356] "Generic (PLEG): container finished" podID="577524c4-7e26-4dd1-9cdb-0161c2690c34" containerID="918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35" exitCode=0 Jul 21 21:50:21.950046 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.950009 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-84f95887c8-npd92" Jul 21 21:50:21.950046 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.950035 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f95887c8-npd92" event={"ID":"577524c4-7e26-4dd1-9cdb-0161c2690c34","Type":"ContainerDied","Data":"918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35"} Jul 21 21:50:21.950620 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.950085 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-84f95887c8-npd92" event={"ID":"577524c4-7e26-4dd1-9cdb-0161c2690c34","Type":"ContainerDied","Data":"641cd5f16fbca49db2dd3c92a7b92f65639c43c02731045ecd09443785045059"} Jul 21 21:50:21.950620 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.950139 2557 scope.go:117] "RemoveContainer" containerID="918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35" Jul 21 21:50:21.958918 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.958899 2557 scope.go:117] "RemoveContainer" containerID="918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35" Jul 21 21:50:21.959287 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:50:21.959242 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35\": container with ID starting with 918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35 not found: ID does not exist" containerID="918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35" Jul 21 21:50:21.959380 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.959295 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35"} err="failed to get container status \"918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35\": rpc error: code = NotFound desc = could not find container \"918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35\": container with ID starting with 918fbcf6e4dfd610c47f4280f676cebb227cc99abe9afb403115dc02a4a46a35 not found: ID does not exist" Jul 21 21:50:21.972289 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.972258 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-84f95887c8-npd92"] Jul 21 21:50:21.978273 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:21.978246 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-84f95887c8-npd92"] Jul 21 21:50:22.389195 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:22.389163 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="577524c4-7e26-4dd1-9cdb-0161c2690c34" path="/var/lib/kubelet/pods/577524c4-7e26-4dd1-9cdb-0161c2690c34/volumes" Jul 21 21:50:22.955933 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:22.955898 2557 generic.go:356] "Generic (PLEG): container finished" podID="1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b" containerID="cb080bc420aa31b6e4cc7bdb903bbb08e777cd57046a63012f2a8474168606f7" exitCode=0 Jul 21 21:50:22.956369 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:22.955971 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" event={"ID":"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b","Type":"ContainerDied","Data":"cb080bc420aa31b6e4cc7bdb903bbb08e777cd57046a63012f2a8474168606f7"} Jul 21 21:50:22.956369 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:22.956354 2557 scope.go:117] "RemoveContainer" containerID="cb080bc420aa31b6e4cc7bdb903bbb08e777cd57046a63012f2a8474168606f7" Jul 21 21:50:23.961803 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:23.961761 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5bcbc94c78-kq4b6" event={"ID":"1dc571c7-3bd1-4c91-b5cd-d1fdb74e3a3b","Type":"ContainerStarted","Data":"94726a6fe491bfe2549ad532d822edb935e8a95931fd5b012942a8ca8e05f225"} Jul 21 21:50:26.120228 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.120188 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" podUID="2db5177f-c854-4edb-85b1-27c573fc9673" containerName="registry" containerID="cri-o://add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc" gracePeriod=30 Jul 21 21:50:26.391758 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.391689 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:50:26.544601 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.544563 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-registry-certificates\") pod \"2db5177f-c854-4edb-85b1-27c573fc9673\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " Jul 21 21:50:26.544601 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.544607 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-installation-pull-secrets\") pod \"2db5177f-c854-4edb-85b1-27c573fc9673\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " Jul 21 21:50:26.544846 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.544629 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h7th2\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-kube-api-access-h7th2\") pod \"2db5177f-c854-4edb-85b1-27c573fc9673\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " Jul 21 21:50:26.544846 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.544668 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-bound-sa-token\") pod \"2db5177f-c854-4edb-85b1-27c573fc9673\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " Jul 21 21:50:26.544846 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.544700 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") pod \"2db5177f-c854-4edb-85b1-27c573fc9673\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " Jul 21 21:50:26.544846 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.544727 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2db5177f-c854-4edb-85b1-27c573fc9673-ca-trust-extracted\") pod \"2db5177f-c854-4edb-85b1-27c573fc9673\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " Jul 21 21:50:26.544846 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.544764 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-image-registry-private-configuration\") pod \"2db5177f-c854-4edb-85b1-27c573fc9673\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " Jul 21 21:50:26.544846 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.544800 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-trusted-ca\") pod \"2db5177f-c854-4edb-85b1-27c573fc9673\" (UID: \"2db5177f-c854-4edb-85b1-27c573fc9673\") " Jul 21 21:50:26.545232 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.545208 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "2db5177f-c854-4edb-85b1-27c573fc9673" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:26.545432 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.545404 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "2db5177f-c854-4edb-85b1-27c573fc9673" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:26.547539 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.547505 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "2db5177f-c854-4edb-85b1-27c573fc9673" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:26.547763 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.547716 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "2db5177f-c854-4edb-85b1-27c573fc9673" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:26.547922 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.547889 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "2db5177f-c854-4edb-85b1-27c573fc9673" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:26.548036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.547916 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "2db5177f-c854-4edb-85b1-27c573fc9673" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:26.548123 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.548042 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-kube-api-access-h7th2" (OuterVolumeSpecName: "kube-api-access-h7th2") pod "2db5177f-c854-4edb-85b1-27c573fc9673" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673"). InnerVolumeSpecName "kube-api-access-h7th2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:26.554376 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.554342 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2db5177f-c854-4edb-85b1-27c573fc9673-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "2db5177f-c854-4edb-85b1-27c573fc9673" (UID: "2db5177f-c854-4edb-85b1-27c573fc9673"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:50:26.645877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.645787 2557 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-registry-tls\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:26.645877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.645820 2557 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/2db5177f-c854-4edb-85b1-27c573fc9673-ca-trust-extracted\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:26.645877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.645831 2557 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-image-registry-private-configuration\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:26.645877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.645840 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-trusted-ca\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:26.645877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.645848 2557 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/2db5177f-c854-4edb-85b1-27c573fc9673-registry-certificates\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:26.645877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.645857 2557 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/2db5177f-c854-4edb-85b1-27c573fc9673-installation-pull-secrets\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:26.645877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.645865 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h7th2\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-kube-api-access-h7th2\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:26.645877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.645874 2557 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2db5177f-c854-4edb-85b1-27c573fc9673-bound-sa-token\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:26.974104 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.974069 2557 generic.go:356] "Generic (PLEG): container finished" podID="2db5177f-c854-4edb-85b1-27c573fc9673" containerID="add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc" exitCode=0 Jul 21 21:50:26.974293 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.974133 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" Jul 21 21:50:26.974293 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.974164 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" event={"ID":"2db5177f-c854-4edb-85b1-27c573fc9673","Type":"ContainerDied","Data":"add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc"} Jul 21 21:50:26.974293 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.974229 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cfc58fd6d-hhtvl" event={"ID":"2db5177f-c854-4edb-85b1-27c573fc9673","Type":"ContainerDied","Data":"3b3c7d8a51cad23a8ef252373272b016f710fdda74262724a7d680af90da7415"} Jul 21 21:50:26.974293 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.974257 2557 scope.go:117] "RemoveContainer" containerID="add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc" Jul 21 21:50:26.983253 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.983237 2557 scope.go:117] "RemoveContainer" containerID="add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc" Jul 21 21:50:26.983558 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:50:26.983539 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc\": container with ID starting with add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc not found: ID does not exist" containerID="add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc" Jul 21 21:50:26.983608 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.983572 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc"} err="failed to get container status \"add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc\": rpc error: code = NotFound desc = could not find container \"add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc\": container with ID starting with add3daf44ebb18eaec321404038d8dd67b5a10b8f79ea2cb207dc78b638458dc not found: ID does not exist" Jul 21 21:50:26.996671 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:26.996636 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5cfc58fd6d-hhtvl"] Jul 21 21:50:27.001509 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:27.001482 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5cfc58fd6d-hhtvl"] Jul 21 21:50:28.389142 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:28.389087 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2db5177f-c854-4edb-85b1-27c573fc9673" path="/var/lib/kubelet/pods/2db5177f-c854-4edb-85b1-27c573fc9673/volumes" Jul 21 21:50:30.278768 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:30.278723 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:50:30.281302 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:30.281277 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4eef30ff-e679-4d2e-bf2b-121e202bf436-metrics-certs\") pod \"network-metrics-daemon-9fdk4\" (UID: \"4eef30ff-e679-4d2e-bf2b-121e202bf436\") " pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:50:30.395681 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:30.395652 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-n9lxs\"" Jul 21 21:50:30.403788 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:30.403763 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9fdk4" Jul 21 21:50:30.568947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:30.568868 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9fdk4"] Jul 21 21:50:30.573039 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:50:30.573007 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4eef30ff_e679_4d2e_bf2b_121e202bf436.slice/crio-c8607d588f689e38e143f7c3abbc51eb9e0fc140a98fc8883e7a6586977d7bcb WatchSource:0}: Error finding container c8607d588f689e38e143f7c3abbc51eb9e0fc140a98fc8883e7a6586977d7bcb: Status 404 returned error can't find the container with id c8607d588f689e38e143f7c3abbc51eb9e0fc140a98fc8883e7a6586977d7bcb Jul 21 21:50:30.991396 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:30.991362 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fdk4" event={"ID":"4eef30ff-e679-4d2e-bf2b-121e202bf436","Type":"ContainerStarted","Data":"c8607d588f689e38e143f7c3abbc51eb9e0fc140a98fc8883e7a6586977d7bcb"} Jul 21 21:50:33.000882 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:33.000849 2557 generic.go:356] "Generic (PLEG): container finished" podID="794afb78-cd2f-4061-8a6b-950b5c00240f" containerID="7249bc45ff43806e3420ea1e87064981958347ccc627edda571074e37065c923" exitCode=0 Jul 21 21:50:33.001358 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:33.000918 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" event={"ID":"794afb78-cd2f-4061-8a6b-950b5c00240f","Type":"ContainerDied","Data":"7249bc45ff43806e3420ea1e87064981958347ccc627edda571074e37065c923"} Jul 21 21:50:33.001358 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:33.001321 2557 scope.go:117] "RemoveContainer" containerID="7249bc45ff43806e3420ea1e87064981958347ccc627edda571074e37065c923" Jul 21 21:50:33.002868 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:33.002839 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fdk4" event={"ID":"4eef30ff-e679-4d2e-bf2b-121e202bf436","Type":"ContainerStarted","Data":"db8f091049ff6b65c900762d76544815bccff4c834326e7b3fb8b6ed881e5d90"} Jul 21 21:50:33.002957 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:33.002884 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9fdk4" event={"ID":"4eef30ff-e679-4d2e-bf2b-121e202bf436","Type":"ContainerStarted","Data":"dbe038c3af7b78f3ad2cc37cc7cd4a452f5ff6c27478f8198d1bccd1db84e2ad"} Jul 21 21:50:33.034755 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:33.034696 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9fdk4" podStartSLOduration=130.627905902 podStartE2EDuration="2m12.034673647s" totalImagesPullingTime="1.406767745s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:48:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:50:33.033604767 +0000 UTC m=+133.139568914" watchObservedRunningTime="2026-07-21 21:50:33.034673647 +0000 UTC m=+133.140637798" Jul 21 21:50:34.011528 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:34.011481 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-7658d9b749-smvjt" event={"ID":"794afb78-cd2f-4061-8a6b-950b5c00240f","Type":"ContainerStarted","Data":"da512397f8f3f14af953af1270ba4798309c8dfacb36d9d260560845cfb7d7c0"} Jul 21 21:50:35.017161 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.017128 2557 generic.go:356] "Generic (PLEG): container finished" podID="25671a47-f968-4834-8534-1f0e3e92f43b" containerID="a2bd3b139b4ac6d3303d3d4f248fc7db4a920ac9707c7e0518ddec3743de41de" exitCode=0 Jul 21 21:50:35.017553 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.017195 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2hqv9" event={"ID":"25671a47-f968-4834-8534-1f0e3e92f43b","Type":"ContainerDied","Data":"a2bd3b139b4ac6d3303d3d4f248fc7db4a920ac9707c7e0518ddec3743de41de"} Jul 21 21:50:35.017553 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.017541 2557 scope.go:117] "RemoveContainer" containerID="a2bd3b139b4ac6d3303d3d4f248fc7db4a920ac9707c7e0518ddec3743de41de" Jul 21 21:50:35.488053 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.488003 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-746fcd7868-lw76b" podUID="cc921385-8886-4469-b9af-134121cb928d" containerName="console" containerID="cri-o://9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f" gracePeriod=15 Jul 21 21:50:35.767473 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.767438 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:35.826577 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.826541 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-console-config\") pod \"cc921385-8886-4469-b9af-134121cb928d\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " Jul 21 21:50:35.826757 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.826591 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-oauth-serving-cert\") pod \"cc921385-8886-4469-b9af-134121cb928d\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " Jul 21 21:50:35.826757 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.826622 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-trusted-ca-bundle\") pod \"cc921385-8886-4469-b9af-134121cb928d\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " Jul 21 21:50:35.826757 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.826641 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-oauth-config\") pod \"cc921385-8886-4469-b9af-134121cb928d\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " Jul 21 21:50:35.826757 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.826671 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-serving-cert\") pod \"cc921385-8886-4469-b9af-134121cb928d\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " Jul 21 21:50:35.826961 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.826797 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-service-ca\") pod \"cc921385-8886-4469-b9af-134121cb928d\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " Jul 21 21:50:35.826961 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.826865 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k69nj\" (UniqueName: \"kubernetes.io/projected/cc921385-8886-4469-b9af-134121cb928d-kube-api-access-k69nj\") pod \"cc921385-8886-4469-b9af-134121cb928d\" (UID: \"cc921385-8886-4469-b9af-134121cb928d\") " Jul 21 21:50:35.827063 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.826953 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "cc921385-8886-4469-b9af-134121cb928d" (UID: "cc921385-8886-4469-b9af-134121cb928d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:35.827172 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.827150 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-oauth-serving-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:35.827223 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.827167 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "cc921385-8886-4469-b9af-134121cb928d" (UID: "cc921385-8886-4469-b9af-134121cb928d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:35.827279 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.827259 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-service-ca" (OuterVolumeSpecName: "service-ca") pod "cc921385-8886-4469-b9af-134121cb928d" (UID: "cc921385-8886-4469-b9af-134121cb928d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:35.827333 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.827292 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-console-config" (OuterVolumeSpecName: "console-config") pod "cc921385-8886-4469-b9af-134121cb928d" (UID: "cc921385-8886-4469-b9af-134121cb928d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:50:35.828942 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.828921 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "cc921385-8886-4469-b9af-134121cb928d" (UID: "cc921385-8886-4469-b9af-134121cb928d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:35.829054 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.829006 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "cc921385-8886-4469-b9af-134121cb928d" (UID: "cc921385-8886-4469-b9af-134121cb928d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:50:35.829054 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.829006 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc921385-8886-4469-b9af-134121cb928d-kube-api-access-k69nj" (OuterVolumeSpecName: "kube-api-access-k69nj") pod "cc921385-8886-4469-b9af-134121cb928d" (UID: "cc921385-8886-4469-b9af-134121cb928d"). InnerVolumeSpecName "kube-api-access-k69nj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:50:35.927674 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.927632 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-serving-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:35.927674 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.927663 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-service-ca\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:35.927674 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.927673 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69nj\" (UniqueName: \"kubernetes.io/projected/cc921385-8886-4469-b9af-134121cb928d-kube-api-access-k69nj\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:35.927674 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.927683 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-console-config\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:35.927941 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.927692 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cc921385-8886-4469-b9af-134121cb928d-trusted-ca-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:35.927941 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:35.927703 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/cc921385-8886-4469-b9af-134121cb928d-console-oauth-config\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:50:36.023069 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.022974 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-2hqv9" event={"ID":"25671a47-f968-4834-8534-1f0e3e92f43b","Type":"ContainerStarted","Data":"b1f0082cf49aad27c0de6a8872d9d2afc0777b40eeb8ff7de5bb79370676df5d"} Jul 21 21:50:36.024201 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.024180 2557 generic.go:356] "Generic (PLEG): container finished" podID="cc921385-8886-4469-b9af-134121cb928d" containerID="9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f" exitCode=0 Jul 21 21:50:36.024301 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.024219 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746fcd7868-lw76b" event={"ID":"cc921385-8886-4469-b9af-134121cb928d","Type":"ContainerDied","Data":"9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f"} Jul 21 21:50:36.024301 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.024225 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-746fcd7868-lw76b" Jul 21 21:50:36.024301 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.024249 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-746fcd7868-lw76b" event={"ID":"cc921385-8886-4469-b9af-134121cb928d","Type":"ContainerDied","Data":"c4b876b3ce9b4b20ee73a53bb3cb1c9ad5f8b6bf2fdf9af16dd078cec969086e"} Jul 21 21:50:36.024301 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.024268 2557 scope.go:117] "RemoveContainer" containerID="9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f" Jul 21 21:50:36.037518 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.037496 2557 scope.go:117] "RemoveContainer" containerID="9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f" Jul 21 21:50:36.037835 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:50:36.037816 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f\": container with ID starting with 9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f not found: ID does not exist" containerID="9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f" Jul 21 21:50:36.037876 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.037849 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f"} err="failed to get container status \"9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f\": rpc error: code = NotFound desc = could not find container \"9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f\": container with ID starting with 9d8d3df94e58892e99435d648ee3595778e7bd1fe2bdd1b19960d9c846f3097f not found: ID does not exist" Jul 21 21:50:36.054833 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.054802 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-746fcd7868-lw76b"] Jul 21 21:50:36.061525 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.061494 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-746fcd7868-lw76b"] Jul 21 21:50:36.388433 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:50:36.388358 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc921385-8886-4469-b9af-134121cb928d" path="/var/lib/kubelet/pods/cc921385-8886-4469-b9af-134121cb928d/volumes" Jul 21 21:51:29.620444 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620405 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7fbc76f785-vv87m"] Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620707 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc921385-8886-4469-b9af-134121cb928d" containerName="console" Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620721 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc921385-8886-4469-b9af-134121cb928d" containerName="console" Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620753 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2db5177f-c854-4edb-85b1-27c573fc9673" containerName="registry" Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620760 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="2db5177f-c854-4edb-85b1-27c573fc9673" containerName="registry" Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620777 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577524c4-7e26-4dd1-9cdb-0161c2690c34" containerName="console" Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620785 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="577524c4-7e26-4dd1-9cdb-0161c2690c34" containerName="console" Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620842 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="2db5177f-c854-4edb-85b1-27c573fc9673" containerName="registry" Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620850 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="577524c4-7e26-4dd1-9cdb-0161c2690c34" containerName="console" Jul 21 21:51:29.620878 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.620856 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc921385-8886-4469-b9af-134121cb928d" containerName="console" Jul 21 21:51:29.625147 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.625122 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.637195 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.637170 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbc76f785-vv87m"] Jul 21 21:51:29.744653 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.744619 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-oauth-serving-cert\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.744853 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.744670 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrlff\" (UniqueName: \"kubernetes.io/projected/f87a0744-f78c-4faa-af31-a92c52d9067f-kube-api-access-nrlff\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.744853 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.744702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-service-ca\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.744853 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.744736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-trusted-ca-bundle\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.744853 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.744832 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-serving-cert\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.745013 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.744866 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-oauth-config\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.745013 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.744882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-console-config\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846038 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.845999 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-serving-cert\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846038 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846041 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-oauth-config\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846371 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846062 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-console-config\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846371 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846123 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-oauth-serving-cert\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846371 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846162 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrlff\" (UniqueName: \"kubernetes.io/projected/f87a0744-f78c-4faa-af31-a92c52d9067f-kube-api-access-nrlff\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846371 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846181 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-service-ca\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846371 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846195 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-trusted-ca-bundle\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846976 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846950 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-console-config\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.846976 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846970 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-oauth-serving-cert\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.847138 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.846958 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-service-ca\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.847579 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.847557 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-trusted-ca-bundle\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.848752 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.848731 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-serving-cert\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.848837 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.848784 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-oauth-config\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.854821 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.854792 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrlff\" (UniqueName: \"kubernetes.io/projected/f87a0744-f78c-4faa-af31-a92c52d9067f-kube-api-access-nrlff\") pod \"console-7fbc76f785-vv87m\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:29.934837 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:29.934742 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:30.069395 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:30.069191 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7fbc76f785-vv87m"] Jul 21 21:51:30.072270 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:51:30.072232 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87a0744_f78c_4faa_af31_a92c52d9067f.slice/crio-8eae546432bbd7eed8bf7a2b2be88b45b0acabe94c5d2938bf86dd099e499819 WatchSource:0}: Error finding container 8eae546432bbd7eed8bf7a2b2be88b45b0acabe94c5d2938bf86dd099e499819: Status 404 returned error can't find the container with id 8eae546432bbd7eed8bf7a2b2be88b45b0acabe94c5d2938bf86dd099e499819 Jul 21 21:51:30.231250 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:30.231212 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbc76f785-vv87m" event={"ID":"f87a0744-f78c-4faa-af31-a92c52d9067f","Type":"ContainerStarted","Data":"01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0"} Jul 21 21:51:30.231416 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:30.231273 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbc76f785-vv87m" event={"ID":"f87a0744-f78c-4faa-af31-a92c52d9067f","Type":"ContainerStarted","Data":"8eae546432bbd7eed8bf7a2b2be88b45b0acabe94c5d2938bf86dd099e499819"} Jul 21 21:51:30.251056 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:30.251008 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7fbc76f785-vv87m" podStartSLOduration=1.250991833 podStartE2EDuration="1.250991833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:51:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:51:30.249504429 +0000 UTC m=+190.355468597" watchObservedRunningTime="2026-07-21 21:51:30.250991833 +0000 UTC m=+190.356955978" Jul 21 21:51:39.935487 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:39.935432 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:39.935487 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:39.935509 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:39.940216 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:39.940192 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:40.277027 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:40.276998 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:51:40.327416 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:51:40.327372 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7996885d64-2kpnv"] Jul 21 21:52:05.358646 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.358582 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7996885d64-2kpnv" podUID="339566fd-1984-405a-9d13-8f9b5cf0bd50" containerName="console" containerID="cri-o://af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae" gracePeriod=15 Jul 21 21:52:05.604606 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.604580 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:52:05.628441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628351 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-oauth-serving-cert\") pod \"339566fd-1984-405a-9d13-8f9b5cf0bd50\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " Jul 21 21:52:05.628441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628394 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-trusted-ca-bundle\") pod \"339566fd-1984-405a-9d13-8f9b5cf0bd50\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " Jul 21 21:52:05.628441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628415 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-config\") pod \"339566fd-1984-405a-9d13-8f9b5cf0bd50\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " Jul 21 21:52:05.628441 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628435 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-service-ca\") pod \"339566fd-1984-405a-9d13-8f9b5cf0bd50\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " Jul 21 21:52:05.628774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628516 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-oauth-config\") pod \"339566fd-1984-405a-9d13-8f9b5cf0bd50\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " Jul 21 21:52:05.628774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628552 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6mqm\" (UniqueName: \"kubernetes.io/projected/339566fd-1984-405a-9d13-8f9b5cf0bd50-kube-api-access-q6mqm\") pod \"339566fd-1984-405a-9d13-8f9b5cf0bd50\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " Jul 21 21:52:05.628774 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628621 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-serving-cert\") pod \"339566fd-1984-405a-9d13-8f9b5cf0bd50\" (UID: \"339566fd-1984-405a-9d13-8f9b5cf0bd50\") " Jul 21 21:52:05.628932 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628819 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "339566fd-1984-405a-9d13-8f9b5cf0bd50" (UID: "339566fd-1984-405a-9d13-8f9b5cf0bd50"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:52:05.628932 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628829 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-config" (OuterVolumeSpecName: "console-config") pod "339566fd-1984-405a-9d13-8f9b5cf0bd50" (UID: "339566fd-1984-405a-9d13-8f9b5cf0bd50"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:52:05.628932 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628833 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "339566fd-1984-405a-9d13-8f9b5cf0bd50" (UID: "339566fd-1984-405a-9d13-8f9b5cf0bd50"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:52:05.628932 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.628845 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-service-ca" (OuterVolumeSpecName: "service-ca") pod "339566fd-1984-405a-9d13-8f9b5cf0bd50" (UID: "339566fd-1984-405a-9d13-8f9b5cf0bd50"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:52:05.629182 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.629115 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-oauth-serving-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:05.629182 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.629141 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-trusted-ca-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:05.629182 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.629156 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-config\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:05.629182 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.629169 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/339566fd-1984-405a-9d13-8f9b5cf0bd50-service-ca\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:05.631036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.631002 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "339566fd-1984-405a-9d13-8f9b5cf0bd50" (UID: "339566fd-1984-405a-9d13-8f9b5cf0bd50"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:52:05.631036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.631019 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "339566fd-1984-405a-9d13-8f9b5cf0bd50" (UID: "339566fd-1984-405a-9d13-8f9b5cf0bd50"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:52:05.631255 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.631062 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/339566fd-1984-405a-9d13-8f9b5cf0bd50-kube-api-access-q6mqm" (OuterVolumeSpecName: "kube-api-access-q6mqm") pod "339566fd-1984-405a-9d13-8f9b5cf0bd50" (UID: "339566fd-1984-405a-9d13-8f9b5cf0bd50"). InnerVolumeSpecName "kube-api-access-q6mqm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:52:05.730488 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.730457 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-oauth-config\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:05.730488 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.730485 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6mqm\" (UniqueName: \"kubernetes.io/projected/339566fd-1984-405a-9d13-8f9b5cf0bd50-kube-api-access-q6mqm\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:05.730488 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:05.730496 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/339566fd-1984-405a-9d13-8f9b5cf0bd50-console-serving-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:06.373849 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.373808 2557 generic.go:356] "Generic (PLEG): container finished" podID="339566fd-1984-405a-9d13-8f9b5cf0bd50" containerID="af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae" exitCode=0 Jul 21 21:52:06.374315 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.373854 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7996885d64-2kpnv" Jul 21 21:52:06.374315 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.373860 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7996885d64-2kpnv" event={"ID":"339566fd-1984-405a-9d13-8f9b5cf0bd50","Type":"ContainerDied","Data":"af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae"} Jul 21 21:52:06.374315 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.373908 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7996885d64-2kpnv" event={"ID":"339566fd-1984-405a-9d13-8f9b5cf0bd50","Type":"ContainerDied","Data":"f12d796ea1aa7c44f973fa550d6fed594f9f6a5cd9391a24f9ff55ef872a6125"} Jul 21 21:52:06.374315 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.373930 2557 scope.go:117] "RemoveContainer" containerID="af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae" Jul 21 21:52:06.383184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.383165 2557 scope.go:117] "RemoveContainer" containerID="af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae" Jul 21 21:52:06.383488 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:52:06.383467 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae\": container with ID starting with af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae not found: ID does not exist" containerID="af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae" Jul 21 21:52:06.383585 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.383495 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae"} err="failed to get container status \"af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae\": rpc error: code = NotFound desc = could not find container \"af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae\": container with ID starting with af20a9e865c625481f0d1d9368b2eed10ab8335cc65e6c4df5a49ab0fade5aae not found: ID does not exist" Jul 21 21:52:06.394112 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.394070 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7996885d64-2kpnv"] Jul 21 21:52:06.398264 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:06.398235 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7996885d64-2kpnv"] Jul 21 21:52:08.388914 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:08.388865 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="339566fd-1984-405a-9d13-8f9b5cf0bd50" path="/var/lib/kubelet/pods/339566fd-1984-405a-9d13-8f9b5cf0bd50/volumes" Jul 21 21:52:32.070036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.070000 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j"] Jul 21 21:52:32.070445 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.070311 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="339566fd-1984-405a-9d13-8f9b5cf0bd50" containerName="console" Jul 21 21:52:32.070445 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.070323 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="339566fd-1984-405a-9d13-8f9b5cf0bd50" containerName="console" Jul 21 21:52:32.070445 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.070385 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="339566fd-1984-405a-9d13-8f9b5cf0bd50" containerName="console" Jul 21 21:52:32.074707 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.074686 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.077469 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.077442 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 21:52:32.077616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.077480 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 21:52:32.077616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.077489 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8mk4r\"" Jul 21 21:52:32.083222 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.083197 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j"] Jul 21 21:52:32.146244 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.146209 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.146424 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.146266 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.146424 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.146380 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlvpq\" (UniqueName: \"kubernetes.io/projected/bc013162-a677-43a6-8bcd-09a1e792338f-kube-api-access-rlvpq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.247540 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.247492 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.247722 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.247562 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlvpq\" (UniqueName: \"kubernetes.io/projected/bc013162-a677-43a6-8bcd-09a1e792338f-kube-api-access-rlvpq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.247722 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.247589 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.247906 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.247885 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.247947 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.247903 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.257577 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.257542 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlvpq\" (UniqueName: \"kubernetes.io/projected/bc013162-a677-43a6-8bcd-09a1e792338f-kube-api-access-rlvpq\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.385475 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.385391 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:32.526535 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:32.526505 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j"] Jul 21 21:52:32.530022 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:52:32.529996 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc013162_a677_43a6_8bcd_09a1e792338f.slice/crio-e6db088ceb1cb9e6eccae679190081e61e46f8802684b7068fea32d8a2084e72 WatchSource:0}: Error finding container e6db088ceb1cb9e6eccae679190081e61e46f8802684b7068fea32d8a2084e72: Status 404 returned error can't find the container with id e6db088ceb1cb9e6eccae679190081e61e46f8802684b7068fea32d8a2084e72 Jul 21 21:52:33.476233 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:33.476187 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" event={"ID":"bc013162-a677-43a6-8bcd-09a1e792338f","Type":"ContainerStarted","Data":"e6db088ceb1cb9e6eccae679190081e61e46f8802684b7068fea32d8a2084e72"} Jul 21 21:52:39.500234 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:39.500195 2557 generic.go:356] "Generic (PLEG): container finished" podID="bc013162-a677-43a6-8bcd-09a1e792338f" containerID="792af04316ace417fce27f90e1419219783f249839a92bcd9201b367fb83e20a" exitCode=0 Jul 21 21:52:39.500777 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:39.500291 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" event={"ID":"bc013162-a677-43a6-8bcd-09a1e792338f","Type":"ContainerDied","Data":"792af04316ace417fce27f90e1419219783f249839a92bcd9201b367fb83e20a"} Jul 21 21:52:42.515575 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:42.515543 2557 generic.go:356] "Generic (PLEG): container finished" podID="bc013162-a677-43a6-8bcd-09a1e792338f" containerID="03c6d870f5f45dd06896ee5bf7fdf91ed41d2fcb6ba3527351bcc972791f9aa9" exitCode=0 Jul 21 21:52:42.515967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:42.515590 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" event={"ID":"bc013162-a677-43a6-8bcd-09a1e792338f","Type":"ContainerDied","Data":"03c6d870f5f45dd06896ee5bf7fdf91ed41d2fcb6ba3527351bcc972791f9aa9"} Jul 21 21:52:50.551755 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:50.551710 2557 generic.go:356] "Generic (PLEG): container finished" podID="bc013162-a677-43a6-8bcd-09a1e792338f" containerID="ea7039be92dd393fa4739beafb931ee2ff69af593f6c2d9eb2a4e7076a8f915f" exitCode=0 Jul 21 21:52:50.552173 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:50.551797 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" event={"ID":"bc013162-a677-43a6-8bcd-09a1e792338f","Type":"ContainerDied","Data":"ea7039be92dd393fa4739beafb931ee2ff69af593f6c2d9eb2a4e7076a8f915f"} Jul 21 21:52:51.677823 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.677794 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:51.813169 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.813047 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rlvpq\" (UniqueName: \"kubernetes.io/projected/bc013162-a677-43a6-8bcd-09a1e792338f-kube-api-access-rlvpq\") pod \"bc013162-a677-43a6-8bcd-09a1e792338f\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " Jul 21 21:52:51.813337 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.813187 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-bundle\") pod \"bc013162-a677-43a6-8bcd-09a1e792338f\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " Jul 21 21:52:51.813337 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.813228 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-util\") pod \"bc013162-a677-43a6-8bcd-09a1e792338f\" (UID: \"bc013162-a677-43a6-8bcd-09a1e792338f\") " Jul 21 21:52:51.813839 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.813816 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-bundle" (OuterVolumeSpecName: "bundle") pod "bc013162-a677-43a6-8bcd-09a1e792338f" (UID: "bc013162-a677-43a6-8bcd-09a1e792338f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:52:51.815639 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.815613 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc013162-a677-43a6-8bcd-09a1e792338f-kube-api-access-rlvpq" (OuterVolumeSpecName: "kube-api-access-rlvpq") pod "bc013162-a677-43a6-8bcd-09a1e792338f" (UID: "bc013162-a677-43a6-8bcd-09a1e792338f"). InnerVolumeSpecName "kube-api-access-rlvpq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:52:51.818195 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.818163 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-util" (OuterVolumeSpecName: "util") pod "bc013162-a677-43a6-8bcd-09a1e792338f" (UID: "bc013162-a677-43a6-8bcd-09a1e792338f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:52:51.914458 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.914409 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlvpq\" (UniqueName: \"kubernetes.io/projected/bc013162-a677-43a6-8bcd-09a1e792338f-kube-api-access-rlvpq\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:51.914458 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.914455 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:51.914458 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:51.914464 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/bc013162-a677-43a6-8bcd-09a1e792338f-util\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:52:52.561751 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:52.561715 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" event={"ID":"bc013162-a677-43a6-8bcd-09a1e792338f","Type":"ContainerDied","Data":"e6db088ceb1cb9e6eccae679190081e61e46f8802684b7068fea32d8a2084e72"} Jul 21 21:52:52.561751 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:52.561755 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bq49j" Jul 21 21:52:52.561955 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:52.561768 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6db088ceb1cb9e6eccae679190081e61e46f8802684b7068fea32d8a2084e72" Jul 21 21:52:59.837328 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.837288 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d"] Jul 21 21:52:59.837713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.837591 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc013162-a677-43a6-8bcd-09a1e792338f" containerName="pull" Jul 21 21:52:59.837713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.837602 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc013162-a677-43a6-8bcd-09a1e792338f" containerName="pull" Jul 21 21:52:59.837713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.837613 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc013162-a677-43a6-8bcd-09a1e792338f" containerName="extract" Jul 21 21:52:59.837713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.837618 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc013162-a677-43a6-8bcd-09a1e792338f" containerName="extract" Jul 21 21:52:59.837713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.837630 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc013162-a677-43a6-8bcd-09a1e792338f" containerName="util" Jul 21 21:52:59.837713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.837636 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc013162-a677-43a6-8bcd-09a1e792338f" containerName="util" Jul 21 21:52:59.837713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.837695 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc013162-a677-43a6-8bcd-09a1e792338f" containerName="extract" Jul 21 21:52:59.842726 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.842708 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" Jul 21 21:52:59.845156 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.845131 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Jul 21 21:52:59.845280 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.845257 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-flk55\"" Jul 21 21:52:59.845371 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.845258 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:52:59.852690 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.852666 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d"] Jul 21 21:52:59.869993 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.869959 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bq4bt\" (UniqueName: \"kubernetes.io/projected/5d3fb868-cdf7-4205-8db9-80425abad245-kube-api-access-bq4bt\") pod \"cert-manager-operator-controller-manager-55c444bf86-xnz7d\" (UID: \"5d3fb868-cdf7-4205-8db9-80425abad245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" Jul 21 21:52:59.869993 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.870002 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3fb868-cdf7-4205-8db9-80425abad245-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xnz7d\" (UID: \"5d3fb868-cdf7-4205-8db9-80425abad245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" Jul 21 21:52:59.970397 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.970355 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bq4bt\" (UniqueName: \"kubernetes.io/projected/5d3fb868-cdf7-4205-8db9-80425abad245-kube-api-access-bq4bt\") pod \"cert-manager-operator-controller-manager-55c444bf86-xnz7d\" (UID: \"5d3fb868-cdf7-4205-8db9-80425abad245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" Jul 21 21:52:59.970397 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.970401 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3fb868-cdf7-4205-8db9-80425abad245-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xnz7d\" (UID: \"5d3fb868-cdf7-4205-8db9-80425abad245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" Jul 21 21:52:59.970769 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.970753 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5d3fb868-cdf7-4205-8db9-80425abad245-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-xnz7d\" (UID: \"5d3fb868-cdf7-4205-8db9-80425abad245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" Jul 21 21:52:59.979248 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:52:59.979211 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bq4bt\" (UniqueName: \"kubernetes.io/projected/5d3fb868-cdf7-4205-8db9-80425abad245-kube-api-access-bq4bt\") pod \"cert-manager-operator-controller-manager-55c444bf86-xnz7d\" (UID: \"5d3fb868-cdf7-4205-8db9-80425abad245\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" Jul 21 21:53:00.153159 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:00.153034 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" Jul 21 21:53:00.312117 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:00.312064 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d"] Jul 21 21:53:00.315025 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:53:00.314995 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3fb868_cdf7_4205_8db9_80425abad245.slice/crio-508c8e2ace43b143b648a178604dd7789b1b2dc39a37071196d2f3190badd0ec WatchSource:0}: Error finding container 508c8e2ace43b143b648a178604dd7789b1b2dc39a37071196d2f3190badd0ec: Status 404 returned error can't find the container with id 508c8e2ace43b143b648a178604dd7789b1b2dc39a37071196d2f3190badd0ec Jul 21 21:53:00.591626 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:00.591588 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" event={"ID":"5d3fb868-cdf7-4205-8db9-80425abad245","Type":"ContainerStarted","Data":"508c8e2ace43b143b648a178604dd7789b1b2dc39a37071196d2f3190badd0ec"} Jul 21 21:53:02.602027 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:02.601982 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" event={"ID":"5d3fb868-cdf7-4205-8db9-80425abad245","Type":"ContainerStarted","Data":"fdec11ae59f65e6ca414c0862be2ec99842e65e302c97cac4e09fd9feda5efd0"} Jul 21 21:53:02.624781 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:02.624702 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-xnz7d" podStartSLOduration=1.550158945 podStartE2EDuration="3.624681232s" totalImagesPullingTime="2.074522287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:52:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:53:02.623757915 +0000 UTC m=+282.729722087" watchObservedRunningTime="2026-07-21 21:53:02.624681232 +0000 UTC m=+282.730645380" Jul 21 21:53:03.925227 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:03.925187 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662"] Jul 21 21:53:03.928704 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:03.928687 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:03.933100 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:03.933067 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 21:53:03.933394 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:03.933379 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8mk4r\"" Jul 21 21:53:03.933899 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:03.933884 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 21:53:03.940518 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:03.940489 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662"] Jul 21 21:53:04.006693 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.006648 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.006887 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.006736 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbqlt\" (UniqueName: \"kubernetes.io/projected/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-kube-api-access-kbqlt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.006887 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.006764 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.108109 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.108064 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.108305 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.108194 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbqlt\" (UniqueName: \"kubernetes.io/projected/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-kube-api-access-kbqlt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.108305 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.108234 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.108616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.108587 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.108616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.108606 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.117563 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.117532 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbqlt\" (UniqueName: \"kubernetes.io/projected/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-kube-api-access-kbqlt\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.237970 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.237936 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:04.382439 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.382402 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662"] Jul 21 21:53:04.386029 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:53:04.386003 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b92aa0_0038_49f5_961b_a3d9c0dfefe0.slice/crio-bd70f5631bb8026bf1c0a77a7ee47c29a50bac299b44ea94e8e3ffe35935604e WatchSource:0}: Error finding container bd70f5631bb8026bf1c0a77a7ee47c29a50bac299b44ea94e8e3ffe35935604e: Status 404 returned error can't find the container with id bd70f5631bb8026bf1c0a77a7ee47c29a50bac299b44ea94e8e3ffe35935604e Jul 21 21:53:04.610953 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.610920 2557 generic.go:356] "Generic (PLEG): container finished" podID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerID="d63ef24eeab73b216de099b87d6982f5411f84c6a3a9978c10349d08f58261b1" exitCode=0 Jul 21 21:53:04.611171 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.611017 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" event={"ID":"67b92aa0-0038-49f5-961b-a3d9c0dfefe0","Type":"ContainerDied","Data":"d63ef24eeab73b216de099b87d6982f5411f84c6a3a9978c10349d08f58261b1"} Jul 21 21:53:04.611171 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:04.611070 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" event={"ID":"67b92aa0-0038-49f5-961b-a3d9c0dfefe0","Type":"ContainerStarted","Data":"bd70f5631bb8026bf1c0a77a7ee47c29a50bac299b44ea94e8e3ffe35935604e"} Jul 21 21:53:06.873640 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:06.873599 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7"] Jul 21 21:53:06.876856 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:06.876829 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" Jul 21 21:53:06.879594 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:06.879563 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Jul 21 21:53:06.879594 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:06.879568 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Jul 21 21:53:06.879829 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:06.879660 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-cnb7x\"" Jul 21 21:53:06.890449 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:06.890416 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7"] Jul 21 21:53:06.930009 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:06.929971 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef918e12-09db-4514-a049-e690dd4353db-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k9vj7\" (UID: \"ef918e12-09db-4514-a049-e690dd4353db\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" Jul 21 21:53:06.930223 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:06.930046 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq47c\" (UniqueName: \"kubernetes.io/projected/ef918e12-09db-4514-a049-e690dd4353db-kube-api-access-nq47c\") pod \"cert-manager-cainjector-559cdd8c87-k9vj7\" (UID: \"ef918e12-09db-4514-a049-e690dd4353db\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" Jul 21 21:53:07.030651 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.030604 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq47c\" (UniqueName: \"kubernetes.io/projected/ef918e12-09db-4514-a049-e690dd4353db-kube-api-access-nq47c\") pod \"cert-manager-cainjector-559cdd8c87-k9vj7\" (UID: \"ef918e12-09db-4514-a049-e690dd4353db\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" Jul 21 21:53:07.030804 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.030661 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef918e12-09db-4514-a049-e690dd4353db-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k9vj7\" (UID: \"ef918e12-09db-4514-a049-e690dd4353db\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" Jul 21 21:53:07.040261 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.040234 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ef918e12-09db-4514-a049-e690dd4353db-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-k9vj7\" (UID: \"ef918e12-09db-4514-a049-e690dd4353db\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" Jul 21 21:53:07.040423 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.040321 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq47c\" (UniqueName: \"kubernetes.io/projected/ef918e12-09db-4514-a049-e690dd4353db-kube-api-access-nq47c\") pod \"cert-manager-cainjector-559cdd8c87-k9vj7\" (UID: \"ef918e12-09db-4514-a049-e690dd4353db\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" Jul 21 21:53:07.199654 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.199611 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" Jul 21 21:53:07.344843 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.344768 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7"] Jul 21 21:53:07.346927 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:53:07.346900 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef918e12_09db_4514_a049_e690dd4353db.slice/crio-3ce48fd1e7f6abaab110eb3966a979a61f3226bbb1cf5c52d462b72fcbe28e80 WatchSource:0}: Error finding container 3ce48fd1e7f6abaab110eb3966a979a61f3226bbb1cf5c52d462b72fcbe28e80: Status 404 returned error can't find the container with id 3ce48fd1e7f6abaab110eb3966a979a61f3226bbb1cf5c52d462b72fcbe28e80 Jul 21 21:53:07.630019 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.629929 2557 generic.go:356] "Generic (PLEG): container finished" podID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerID="0de921edddbadea1d454dda29c7c47798773f40a7f4eba6ad498f28ce75859a7" exitCode=0 Jul 21 21:53:07.630188 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.630019 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" event={"ID":"67b92aa0-0038-49f5-961b-a3d9c0dfefe0","Type":"ContainerDied","Data":"0de921edddbadea1d454dda29c7c47798773f40a7f4eba6ad498f28ce75859a7"} Jul 21 21:53:07.631312 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:07.631284 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" event={"ID":"ef918e12-09db-4514-a049-e690dd4353db","Type":"ContainerStarted","Data":"3ce48fd1e7f6abaab110eb3966a979a61f3226bbb1cf5c52d462b72fcbe28e80"} Jul 21 21:53:08.638181 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:08.638140 2557 generic.go:356] "Generic (PLEG): container finished" podID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerID="69ae1656789611d85cd7ddb31afaff86eb587fa9df91762b23378225f303e9bd" exitCode=0 Jul 21 21:53:08.638637 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:08.638230 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" event={"ID":"67b92aa0-0038-49f5-961b-a3d9c0dfefe0","Type":"ContainerDied","Data":"69ae1656789611d85cd7ddb31afaff86eb587fa9df91762b23378225f303e9bd"} Jul 21 21:53:09.779366 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.779337 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:09.853265 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.853231 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-bundle\") pod \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " Jul 21 21:53:09.853435 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.853307 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-util\") pod \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " Jul 21 21:53:09.853435 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.853356 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbqlt\" (UniqueName: \"kubernetes.io/projected/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-kube-api-access-kbqlt\") pod \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\" (UID: \"67b92aa0-0038-49f5-961b-a3d9c0dfefe0\") " Jul 21 21:53:09.853652 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.853630 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-bundle" (OuterVolumeSpecName: "bundle") pod "67b92aa0-0038-49f5-961b-a3d9c0dfefe0" (UID: "67b92aa0-0038-49f5-961b-a3d9c0dfefe0"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:53:09.855692 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.855658 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-kube-api-access-kbqlt" (OuterVolumeSpecName: "kube-api-access-kbqlt") pod "67b92aa0-0038-49f5-961b-a3d9c0dfefe0" (UID: "67b92aa0-0038-49f5-961b-a3d9c0dfefe0"). InnerVolumeSpecName "kube-api-access-kbqlt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:53:09.876949 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.876890 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-util" (OuterVolumeSpecName: "util") pod "67b92aa0-0038-49f5-961b-a3d9c0dfefe0" (UID: "67b92aa0-0038-49f5-961b-a3d9c0dfefe0"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:53:09.954882 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.954785 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-util\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:53:09.954882 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.954817 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbqlt\" (UniqueName: \"kubernetes.io/projected/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-kube-api-access-kbqlt\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:53:09.954882 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:09.954827 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/67b92aa0-0038-49f5-961b-a3d9c0dfefe0-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:53:10.654372 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:10.654268 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" event={"ID":"ef918e12-09db-4514-a049-e690dd4353db","Type":"ContainerStarted","Data":"7d898381434257a65e2680960fd64f114e3472bbee3d9400c5494d1eff56aa98"} Jul 21 21:53:10.656078 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:10.656054 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" event={"ID":"67b92aa0-0038-49f5-961b-a3d9c0dfefe0","Type":"ContainerDied","Data":"bd70f5631bb8026bf1c0a77a7ee47c29a50bac299b44ea94e8e3ffe35935604e"} Jul 21 21:53:10.656222 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:10.656063 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fvz662" Jul 21 21:53:10.656222 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:10.656133 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd70f5631bb8026bf1c0a77a7ee47c29a50bac299b44ea94e8e3ffe35935604e" Jul 21 21:53:10.670479 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:10.670423 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-k9vj7" podStartSLOduration=2.256655812 podStartE2EDuration="4.670405946s" totalImagesPullingTime="2.413750134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:53:10.668879875 +0000 UTC m=+290.774844021" watchObservedRunningTime="2026-07-21 21:53:10.670405946 +0000 UTC m=+290.776370093" Jul 21 21:53:11.348368 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.348326 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ktsrd"] Jul 21 21:53:11.348771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.348638 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerName="pull" Jul 21 21:53:11.348771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.348651 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerName="pull" Jul 21 21:53:11.348771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.348665 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerName="extract" Jul 21 21:53:11.348771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.348670 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerName="extract" Jul 21 21:53:11.348771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.348678 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerName="util" Jul 21 21:53:11.348771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.348684 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerName="util" Jul 21 21:53:11.348771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.348735 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="67b92aa0-0038-49f5-961b-a3d9c0dfefe0" containerName="extract" Jul 21 21:53:11.351498 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.351476 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:11.353650 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.353630 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-sfh4v\"" Jul 21 21:53:11.359870 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.359845 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ktsrd"] Jul 21 21:53:11.466147 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.466065 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ada112-bc5f-45a8-bc20-b336a6935a86-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ktsrd\" (UID: \"b3ada112-bc5f-45a8-bc20-b336a6935a86\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:11.466330 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.466204 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgsb9\" (UniqueName: \"kubernetes.io/projected/b3ada112-bc5f-45a8-bc20-b336a6935a86-kube-api-access-hgsb9\") pod \"cert-manager-webhook-6486ffd997-ktsrd\" (UID: \"b3ada112-bc5f-45a8-bc20-b336a6935a86\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:11.566647 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.566606 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgsb9\" (UniqueName: \"kubernetes.io/projected/b3ada112-bc5f-45a8-bc20-b336a6935a86-kube-api-access-hgsb9\") pod \"cert-manager-webhook-6486ffd997-ktsrd\" (UID: \"b3ada112-bc5f-45a8-bc20-b336a6935a86\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:11.566837 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.566681 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ada112-bc5f-45a8-bc20-b336a6935a86-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ktsrd\" (UID: \"b3ada112-bc5f-45a8-bc20-b336a6935a86\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:11.575602 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.575566 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b3ada112-bc5f-45a8-bc20-b336a6935a86-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-ktsrd\" (UID: \"b3ada112-bc5f-45a8-bc20-b336a6935a86\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:11.575733 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.575718 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgsb9\" (UniqueName: \"kubernetes.io/projected/b3ada112-bc5f-45a8-bc20-b336a6935a86-kube-api-access-hgsb9\") pod \"cert-manager-webhook-6486ffd997-ktsrd\" (UID: \"b3ada112-bc5f-45a8-bc20-b336a6935a86\") " pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:11.660681 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.660596 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:11.805260 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:11.805168 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-ktsrd"] Jul 21 21:53:11.807521 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:53:11.807488 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3ada112_bc5f_45a8_bc20_b336a6935a86.slice/crio-8217083eaa2093658f5411dc64afcd88fabfcc36a64e86f6ff74d3fcf6d14840 WatchSource:0}: Error finding container 8217083eaa2093658f5411dc64afcd88fabfcc36a64e86f6ff74d3fcf6d14840: Status 404 returned error can't find the container with id 8217083eaa2093658f5411dc64afcd88fabfcc36a64e86f6ff74d3fcf6d14840 Jul 21 21:53:12.665863 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:12.665816 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" event={"ID":"b3ada112-bc5f-45a8-bc20-b336a6935a86","Type":"ContainerStarted","Data":"195395cadde69fd753b5a5b1168fb7e1b42532c6c864059975e61f8ea35572ee"} Jul 21 21:53:12.665863 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:12.665865 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" event={"ID":"b3ada112-bc5f-45a8-bc20-b336a6935a86","Type":"ContainerStarted","Data":"8217083eaa2093658f5411dc64afcd88fabfcc36a64e86f6ff74d3fcf6d14840"} Jul 21 21:53:12.666330 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:12.665888 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:12.683329 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:12.683271 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" podStartSLOduration=1.683251673 podStartE2EDuration="1.683251673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:53:12.681592376 +0000 UTC m=+292.787556522" watchObservedRunningTime="2026-07-21 21:53:12.683251673 +0000 UTC m=+292.789215819" Jul 21 21:53:18.672809 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:18.672781 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-ktsrd" Jul 21 21:53:20.219172 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:20.219141 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 21:53:20.221301 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:20.221282 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 21:53:20.229835 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:20.229807 2557 kubelet.go:1648] "Image garbage collection succeeded" Jul 21 21:53:21.368934 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.368897 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f"] Jul 21 21:53:21.372291 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.372275 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.374701 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.374641 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Jul 21 21:53:21.375680 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.375648 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Jul 21 21:53:21.375794 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.375734 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-8mk4r\"" Jul 21 21:53:21.380671 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.380638 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f"] Jul 21 21:53:21.446479 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.446439 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rk8bd\" (UniqueName: \"kubernetes.io/projected/9533ed5d-1c48-4fdc-ae01-eeab66e68013-kube-api-access-rk8bd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.446678 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.446500 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.446678 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.446571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.547629 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.547594 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rk8bd\" (UniqueName: \"kubernetes.io/projected/9533ed5d-1c48-4fdc-ae01-eeab66e68013-kube-api-access-rk8bd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.547798 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.547654 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.547798 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.547702 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.548006 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.547986 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.548055 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.548039 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.557149 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.557120 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rk8bd\" (UniqueName: \"kubernetes.io/projected/9533ed5d-1c48-4fdc-ae01-eeab66e68013-kube-api-access-rk8bd\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.682517 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.682415 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:21.823964 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:21.823939 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f"] Jul 21 21:53:21.826300 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:53:21.826269 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9533ed5d_1c48_4fdc_ae01_eeab66e68013.slice/crio-81cf8320df935f62860180b161dcb5b34fad478df60284fea5f0fe95d9bd47e1 WatchSource:0}: Error finding container 81cf8320df935f62860180b161dcb5b34fad478df60284fea5f0fe95d9bd47e1: Status 404 returned error can't find the container with id 81cf8320df935f62860180b161dcb5b34fad478df60284fea5f0fe95d9bd47e1 Jul 21 21:53:22.710263 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:22.710221 2557 generic.go:356] "Generic (PLEG): container finished" podID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerID="aefc3e37c346b1a7d521b30a54a216becfd4fd8f9f7d44ade36699e34df7ce16" exitCode=0 Jul 21 21:53:22.710648 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:22.710269 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" event={"ID":"9533ed5d-1c48-4fdc-ae01-eeab66e68013","Type":"ContainerDied","Data":"aefc3e37c346b1a7d521b30a54a216becfd4fd8f9f7d44ade36699e34df7ce16"} Jul 21 21:53:22.710648 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:22.710312 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" event={"ID":"9533ed5d-1c48-4fdc-ae01-eeab66e68013","Type":"ContainerStarted","Data":"81cf8320df935f62860180b161dcb5b34fad478df60284fea5f0fe95d9bd47e1"} Jul 21 21:53:22.710792 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:22.710777 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:53:23.716620 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:23.716585 2557 generic.go:356] "Generic (PLEG): container finished" podID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerID="a613846a0a2790ea2a4e3c4435e47a0a374caea8c5b9563cf409be597ffde68c" exitCode=0 Jul 21 21:53:23.716993 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:23.716640 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" event={"ID":"9533ed5d-1c48-4fdc-ae01-eeab66e68013","Type":"ContainerDied","Data":"a613846a0a2790ea2a4e3c4435e47a0a374caea8c5b9563cf409be597ffde68c"} Jul 21 21:53:24.723407 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:24.723367 2557 generic.go:356] "Generic (PLEG): container finished" podID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerID="d166141a8efedd5ff45633744cf9b9c5a78ffdef507cf0bbbe1e586897403772" exitCode=0 Jul 21 21:53:24.723812 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:24.723450 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" event={"ID":"9533ed5d-1c48-4fdc-ae01-eeab66e68013","Type":"ContainerDied","Data":"d166141a8efedd5ff45633744cf9b9c5a78ffdef507cf0bbbe1e586897403772"} Jul 21 21:53:25.858783 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:25.858758 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:25.983970 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:25.983883 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-bundle\") pod \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " Jul 21 21:53:25.983970 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:25.983936 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rk8bd\" (UniqueName: \"kubernetes.io/projected/9533ed5d-1c48-4fdc-ae01-eeab66e68013-kube-api-access-rk8bd\") pod \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " Jul 21 21:53:25.984207 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:25.983997 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-util\") pod \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\" (UID: \"9533ed5d-1c48-4fdc-ae01-eeab66e68013\") " Jul 21 21:53:25.984763 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:25.984644 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-bundle" (OuterVolumeSpecName: "bundle") pod "9533ed5d-1c48-4fdc-ae01-eeab66e68013" (UID: "9533ed5d-1c48-4fdc-ae01-eeab66e68013"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:53:25.986212 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:25.986178 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9533ed5d-1c48-4fdc-ae01-eeab66e68013-kube-api-access-rk8bd" (OuterVolumeSpecName: "kube-api-access-rk8bd") pod "9533ed5d-1c48-4fdc-ae01-eeab66e68013" (UID: "9533ed5d-1c48-4fdc-ae01-eeab66e68013"). InnerVolumeSpecName "kube-api-access-rk8bd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:53:25.992569 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:25.992546 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-util" (OuterVolumeSpecName: "util") pod "9533ed5d-1c48-4fdc-ae01-eeab66e68013" (UID: "9533ed5d-1c48-4fdc-ae01-eeab66e68013"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:53:26.085487 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:26.085451 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-util\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:53:26.085487 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:26.085483 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9533ed5d-1c48-4fdc-ae01-eeab66e68013-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:53:26.085487 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:26.085493 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rk8bd\" (UniqueName: \"kubernetes.io/projected/9533ed5d-1c48-4fdc-ae01-eeab66e68013-kube-api-access-rk8bd\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:53:26.739896 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:26.739858 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" event={"ID":"9533ed5d-1c48-4fdc-ae01-eeab66e68013","Type":"ContainerDied","Data":"81cf8320df935f62860180b161dcb5b34fad478df60284fea5f0fe95d9bd47e1"} Jul 21 21:53:26.740126 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:26.739910 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afrdf4f" Jul 21 21:53:26.740126 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:26.739947 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81cf8320df935f62860180b161dcb5b34fad478df60284fea5f0fe95d9bd47e1" Jul 21 21:53:38.681157 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.681122 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn"] Jul 21 21:53:38.681651 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.681521 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerName="pull" Jul 21 21:53:38.681651 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.681537 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerName="pull" Jul 21 21:53:38.681651 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.681559 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerName="extract" Jul 21 21:53:38.681651 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.681567 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerName="extract" Jul 21 21:53:38.681651 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.681581 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerName="util" Jul 21 21:53:38.681651 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.681589 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerName="util" Jul 21 21:53:38.681955 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.681673 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9533ed5d-1c48-4fdc-ae01-eeab66e68013" containerName="extract" Jul 21 21:53:38.687697 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.687672 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.690541 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.690520 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 21 21:53:38.690739 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.690725 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-fdpkb\"" Jul 21 21:53:38.691138 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.691121 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 21 21:53:38.691221 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.691186 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 21 21:53:38.691290 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.691269 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 21 21:53:38.725474 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.725434 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn"] Jul 21 21:53:38.794324 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.794291 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.794537 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.794345 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.794537 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.794367 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcpxh\" (UniqueName: \"kubernetes.io/projected/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-kube-api-access-fcpxh\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.794537 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.794399 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.794537 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.794416 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.895622 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.895581 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.895622 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.895626 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcpxh\" (UniqueName: \"kubernetes.io/projected/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-kube-api-access-fcpxh\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.895916 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.895647 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.895916 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.895673 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.895916 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.895722 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.896163 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.896138 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-manifests\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.898321 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.898288 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-tmp\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.898493 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.898469 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.898655 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.898639 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-webhook-cert\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.907684 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.907659 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcpxh\" (UniqueName: \"kubernetes.io/projected/63c8eb32-068c-420d-82ca-2a4cbe3cb4ab-kube-api-access-fcpxh\") pod \"opendatahub-operator-controller-manager-54667d85dc-5sfcn\" (UID: \"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab\") " pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:38.998294 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:38.998243 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:39.141716 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:39.141682 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn"] Jul 21 21:53:39.145542 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:53:39.145511 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63c8eb32_068c_420d_82ca_2a4cbe3cb4ab.slice/crio-ed561258f51bd8b8e227c9db83a3555df555e1baccb01c10ddaebdf5ef3bd3b3 WatchSource:0}: Error finding container ed561258f51bd8b8e227c9db83a3555df555e1baccb01c10ddaebdf5ef3bd3b3: Status 404 returned error can't find the container with id ed561258f51bd8b8e227c9db83a3555df555e1baccb01c10ddaebdf5ef3bd3b3 Jul 21 21:53:39.805171 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:39.805116 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" event={"ID":"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab","Type":"ContainerStarted","Data":"ed561258f51bd8b8e227c9db83a3555df555e1baccb01c10ddaebdf5ef3bd3b3"} Jul 21 21:53:42.822942 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:42.822902 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" event={"ID":"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab","Type":"ContainerStarted","Data":"5a04a2898efb6f4c0991b5d1f4bb1c7cead8aed5a0c350e1d1cacaee15e9c995"} Jul 21 21:53:43.828838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:43.828805 2557 generic.go:356] "Generic (PLEG): container finished" podID="63c8eb32-068c-420d-82ca-2a4cbe3cb4ab" containerID="5a04a2898efb6f4c0991b5d1f4bb1c7cead8aed5a0c350e1d1cacaee15e9c995" exitCode=0 Jul 21 21:53:43.829247 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:43.828889 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" event={"ID":"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab","Type":"ContainerDied","Data":"5a04a2898efb6f4c0991b5d1f4bb1c7cead8aed5a0c350e1d1cacaee15e9c995"} Jul 21 21:53:44.835951 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:44.835915 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" event={"ID":"63c8eb32-068c-420d-82ca-2a4cbe3cb4ab","Type":"ContainerStarted","Data":"bd3024b70286999b5e8c30bb09785e69f842be9984863e19c0e3473d4f38b1b0"} Jul 21 21:53:44.836371 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:44.835983 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:44.866843 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:44.866785 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" podStartSLOduration=1.928525616 podStartE2EDuration="6.866768908s" totalImagesPullingTime="3.914325126s" totalInitContainerRuntime="1.023918166s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:38 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:53:44.858640678 +0000 UTC m=+324.964604805" watchObservedRunningTime="2026-07-21 21:53:44.866768908 +0000 UTC m=+324.972733054" Jul 21 21:53:55.842660 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:55.842624 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54667d85dc-5sfcn" Jul 21 21:53:58.745668 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.745624 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n4qw9"] Jul 21 21:53:58.754187 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.754153 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:58.757431 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.757409 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pjj6r\"" Jul 21 21:53:58.758486 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.758466 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Jul 21 21:53:58.765954 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.765926 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9mqp\" (UniqueName: \"kubernetes.io/projected/47ef4f44-ca7b-4104-9041-143b18022eb1-kube-api-access-j9mqp\") pod \"odh-model-controller-7998c67dc4-n4qw9\" (UID: \"47ef4f44-ca7b-4104-9041-143b18022eb1\") " pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:58.766078 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.765969 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ef4f44-ca7b-4104-9041-143b18022eb1-cert\") pod \"odh-model-controller-7998c67dc4-n4qw9\" (UID: \"47ef4f44-ca7b-4104-9041-143b18022eb1\") " pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:58.778236 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.778207 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n4qw9"] Jul 21 21:53:58.866973 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.866934 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9mqp\" (UniqueName: \"kubernetes.io/projected/47ef4f44-ca7b-4104-9041-143b18022eb1-kube-api-access-j9mqp\") pod \"odh-model-controller-7998c67dc4-n4qw9\" (UID: \"47ef4f44-ca7b-4104-9041-143b18022eb1\") " pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:58.867291 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.867268 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ef4f44-ca7b-4104-9041-143b18022eb1-cert\") pod \"odh-model-controller-7998c67dc4-n4qw9\" (UID: \"47ef4f44-ca7b-4104-9041-143b18022eb1\") " pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:58.867439 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:53:58.867407 2557 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found Jul 21 21:53:58.867511 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:53:58.867494 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/47ef4f44-ca7b-4104-9041-143b18022eb1-cert podName:47ef4f44-ca7b-4104-9041-143b18022eb1 nodeName:}" failed. No retries permitted until 2026-07-21 21:53:59.367472256 +0000 UTC m=+339.473436382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/47ef4f44-ca7b-4104-9041-143b18022eb1-cert") pod "odh-model-controller-7998c67dc4-n4qw9" (UID: "47ef4f44-ca7b-4104-9041-143b18022eb1") : secret "odh-model-controller-webhook-cert" not found Jul 21 21:53:58.896332 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:58.896300 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9mqp\" (UniqueName: \"kubernetes.io/projected/47ef4f44-ca7b-4104-9041-143b18022eb1-kube-api-access-j9mqp\") pod \"odh-model-controller-7998c67dc4-n4qw9\" (UID: \"47ef4f44-ca7b-4104-9041-143b18022eb1\") " pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:59.371284 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.371238 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ef4f44-ca7b-4104-9041-143b18022eb1-cert\") pod \"odh-model-controller-7998c67dc4-n4qw9\" (UID: \"47ef4f44-ca7b-4104-9041-143b18022eb1\") " pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:59.374000 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.373975 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/47ef4f44-ca7b-4104-9041-143b18022eb1-cert\") pod \"odh-model-controller-7998c67dc4-n4qw9\" (UID: \"47ef4f44-ca7b-4104-9041-143b18022eb1\") " pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:59.664161 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.664027 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:53:59.685332 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.685294 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-594787b58-x8sdj"] Jul 21 21:53:59.688894 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.688871 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.691840 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.691521 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 21 21:53:59.691840 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.691616 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-k4z8d\"" Jul 21 21:53:59.691840 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.691763 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 21 21:53:59.699527 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.699465 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594787b58-x8sdj"] Jul 21 21:53:59.773637 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.773600 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b67417-ddfd-4dd5-bede-d132bc7752f2-tls-certs\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.774105 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.773654 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b67417-ddfd-4dd5-bede-d132bc7752f2-tmp\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.774105 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.773686 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkf7l\" (UniqueName: \"kubernetes.io/projected/50b67417-ddfd-4dd5-bede-d132bc7752f2-kube-api-access-wkf7l\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.832800 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.832768 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-n4qw9"] Jul 21 21:53:59.834278 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:53:59.834251 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ef4f44_ca7b_4104_9041_143b18022eb1.slice/crio-79a0af8f23f3d7d4909b201f75e317783ad7cf767e1b8b5d11170519a2ac1ed2 WatchSource:0}: Error finding container 79a0af8f23f3d7d4909b201f75e317783ad7cf767e1b8b5d11170519a2ac1ed2: Status 404 returned error can't find the container with id 79a0af8f23f3d7d4909b201f75e317783ad7cf767e1b8b5d11170519a2ac1ed2 Jul 21 21:53:59.874422 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.874384 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b67417-ddfd-4dd5-bede-d132bc7752f2-tmp\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.874611 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.874443 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkf7l\" (UniqueName: \"kubernetes.io/projected/50b67417-ddfd-4dd5-bede-d132bc7752f2-kube-api-access-wkf7l\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.874611 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.874502 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b67417-ddfd-4dd5-bede-d132bc7752f2-tls-certs\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.874685 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:53:59.874629 2557 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Jul 21 21:53:59.874719 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:53:59.874699 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/50b67417-ddfd-4dd5-bede-d132bc7752f2-tls-certs podName:50b67417-ddfd-4dd5-bede-d132bc7752f2 nodeName:}" failed. No retries permitted until 2026-07-21 21:54:00.3746791 +0000 UTC m=+340.480643328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/50b67417-ddfd-4dd5-bede-d132bc7752f2-tls-certs") pod "kube-auth-proxy-594787b58-x8sdj" (UID: "50b67417-ddfd-4dd5-bede-d132bc7752f2") : secret "kube-auth-proxy-tls" not found Jul 21 21:53:59.876836 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.876804 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/50b67417-ddfd-4dd5-bede-d132bc7752f2-tmp\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.887656 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.887628 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkf7l\" (UniqueName: \"kubernetes.io/projected/50b67417-ddfd-4dd5-bede-d132bc7752f2-kube-api-access-wkf7l\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:53:59.905890 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.905853 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" event={"ID":"47ef4f44-ca7b-4104-9041-143b18022eb1","Type":"ContainerStarted","Data":"79a0af8f23f3d7d4909b201f75e317783ad7cf767e1b8b5d11170519a2ac1ed2"} Jul 21 21:53:59.908390 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.908361 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zr6gw"] Jul 21 21:53:59.912788 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.912764 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:53:59.915455 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.915388 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Jul 21 21:53:59.915455 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.915405 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-h49d4\"" Jul 21 21:53:59.924167 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.924136 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zr6gw"] Jul 21 21:53:59.975608 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.975569 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ssfh8\" (UniqueName: \"kubernetes.io/projected/14f720e4-99c1-4618-8be1-28a4f2881372-kube-api-access-ssfh8\") pod \"model-serving-api-54d89bd5fc-zr6gw\" (UID: \"14f720e4-99c1-4618-8be1-28a4f2881372\") " pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:53:59.975794 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:53:59.975642 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f720e4-99c1-4618-8be1-28a4f2881372-tls-certs\") pod \"model-serving-api-54d89bd5fc-zr6gw\" (UID: \"14f720e4-99c1-4618-8be1-28a4f2881372\") " pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:00.077030 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.076977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ssfh8\" (UniqueName: \"kubernetes.io/projected/14f720e4-99c1-4618-8be1-28a4f2881372-kube-api-access-ssfh8\") pod \"model-serving-api-54d89bd5fc-zr6gw\" (UID: \"14f720e4-99c1-4618-8be1-28a4f2881372\") " pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:00.077285 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.077054 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f720e4-99c1-4618-8be1-28a4f2881372-tls-certs\") pod \"model-serving-api-54d89bd5fc-zr6gw\" (UID: \"14f720e4-99c1-4618-8be1-28a4f2881372\") " pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:00.077285 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:54:00.077188 2557 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found Jul 21 21:54:00.077285 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:54:00.077252 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/14f720e4-99c1-4618-8be1-28a4f2881372-tls-certs podName:14f720e4-99c1-4618-8be1-28a4f2881372 nodeName:}" failed. No retries permitted until 2026-07-21 21:54:00.577234051 +0000 UTC m=+340.683198175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/14f720e4-99c1-4618-8be1-28a4f2881372-tls-certs") pod "model-serving-api-54d89bd5fc-zr6gw" (UID: "14f720e4-99c1-4618-8be1-28a4f2881372") : secret "model-serving-api-tls" not found Jul 21 21:54:00.090713 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.090684 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ssfh8\" (UniqueName: \"kubernetes.io/projected/14f720e4-99c1-4618-8be1-28a4f2881372-kube-api-access-ssfh8\") pod \"model-serving-api-54d89bd5fc-zr6gw\" (UID: \"14f720e4-99c1-4618-8be1-28a4f2881372\") " pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:00.379928 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.379888 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b67417-ddfd-4dd5-bede-d132bc7752f2-tls-certs\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:54:00.383158 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.383123 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/50b67417-ddfd-4dd5-bede-d132bc7752f2-tls-certs\") pod \"kube-auth-proxy-594787b58-x8sdj\" (UID: \"50b67417-ddfd-4dd5-bede-d132bc7752f2\") " pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:54:00.582137 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.582068 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f720e4-99c1-4618-8be1-28a4f2881372-tls-certs\") pod \"model-serving-api-54d89bd5fc-zr6gw\" (UID: \"14f720e4-99c1-4618-8be1-28a4f2881372\") " pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:00.585149 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.585117 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14f720e4-99c1-4618-8be1-28a4f2881372-tls-certs\") pod \"model-serving-api-54d89bd5fc-zr6gw\" (UID: \"14f720e4-99c1-4618-8be1-28a4f2881372\") " pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:00.601767 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.601345 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" Jul 21 21:54:00.714877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.714837 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-csc4j"] Jul 21 21:54:00.725748 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.725713 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.729668 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.729031 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Jul 21 21:54:00.729668 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.729327 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Jul 21 21:54:00.729668 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.729467 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Jul 21 21:54:00.729668 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.729654 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-njsh6\"" Jul 21 21:54:00.729982 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.729853 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 21 21:54:00.733234 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.733203 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-csc4j"] Jul 21 21:54:00.784308 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.784269 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f381058e-1c7f-4e93-a9c0-3b875f0500c9-local-certs\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.784807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.784334 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.784807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.784371 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.784807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.784402 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-token\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.784807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.784459 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.784807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.784530 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-cacerts\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.784807 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.784559 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4ldn\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-kube-api-access-z4ldn\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.810335 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.810017 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-594787b58-x8sdj"] Jul 21 21:54:00.814370 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:54:00.814331 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50b67417_ddfd_4dd5_bede_d132bc7752f2.slice/crio-87972d1570b5366b80420d6f3d75b3d0bd4e574218486e4b144ff8fbb1ca206b WatchSource:0}: Error finding container 87972d1570b5366b80420d6f3d75b3d0bd4e574218486e4b144ff8fbb1ca206b: Status 404 returned error can't find the container with id 87972d1570b5366b80420d6f3d75b3d0bd4e574218486e4b144ff8fbb1ca206b Jul 21 21:54:00.825749 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.825588 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:00.885665 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.885629 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.885873 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.885703 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-cacerts\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.885873 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.885736 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4ldn\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-kube-api-access-z4ldn\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.885873 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.885813 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f381058e-1c7f-4e93-a9c0-3b875f0500c9-local-certs\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.885873 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.885851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.886160 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.885879 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.886160 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.885916 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-token\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.886465 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.886438 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.889260 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.889236 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-cacerts\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.889457 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.889437 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f381058e-1c7f-4e93-a9c0-3b875f0500c9-local-certs\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.889749 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.889726 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.890374 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.890349 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.904594 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.904560 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-token\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.908871 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.908821 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4ldn\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-kube-api-access-z4ldn\") pod \"istiod-openshift-gateway-7574459656-csc4j\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:00.923964 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:00.923828 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" event={"ID":"50b67417-ddfd-4dd5-bede-d132bc7752f2","Type":"ContainerStarted","Data":"87972d1570b5366b80420d6f3d75b3d0bd4e574218486e4b144ff8fbb1ca206b"} Jul 21 21:54:01.024871 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:01.024833 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zr6gw"] Jul 21 21:54:01.040031 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:01.039947 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:01.228456 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:01.227719 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-csc4j"] Jul 21 21:54:01.230307 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:54:01.230269 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf381058e_1c7f_4e93_a9c0_3b875f0500c9.slice/crio-897201b44c49d1be561a6aff7d680bc53b419efd950bd3cfb67d07843bfcb3e4 WatchSource:0}: Error finding container 897201b44c49d1be561a6aff7d680bc53b419efd950bd3cfb67d07843bfcb3e4: Status 404 returned error can't find the container with id 897201b44c49d1be561a6aff7d680bc53b419efd950bd3cfb67d07843bfcb3e4 Jul 21 21:54:01.932310 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:01.932066 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" event={"ID":"f381058e-1c7f-4e93-a9c0-3b875f0500c9","Type":"ContainerStarted","Data":"897201b44c49d1be561a6aff7d680bc53b419efd950bd3cfb67d07843bfcb3e4"} Jul 21 21:54:01.933627 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:01.933590 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" event={"ID":"14f720e4-99c1-4618-8be1-28a4f2881372","Type":"ContainerStarted","Data":"8f036e06969e5b3e69fcea7576beb50a36027b21d95d301d2c946c4f6cee504e"} Jul 21 21:54:05.422457 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.422404 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 21 21:54:05.422846 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.422522 2557 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"} Jul 21 21:54:05.964945 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.964898 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" event={"ID":"47ef4f44-ca7b-4104-9041-143b18022eb1","Type":"ContainerStarted","Data":"57e794e5e9c8be45d011215a7876d5b635a0a7b04f9e920598c79e4259a9d89a"} Jul 21 21:54:05.965158 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.965085 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:54:05.966531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.966507 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" event={"ID":"50b67417-ddfd-4dd5-bede-d132bc7752f2","Type":"ContainerStarted","Data":"4a507715ba5243de2adee3af437e0d6b78d60c7bd8caa331593be3784bff4ba3"} Jul 21 21:54:05.968310 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.968284 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" event={"ID":"f381058e-1c7f-4e93-a9c0-3b875f0500c9","Type":"ContainerStarted","Data":"62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4"} Jul 21 21:54:05.968420 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.968329 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:05.969966 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.969943 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" event={"ID":"14f720e4-99c1-4618-8be1-28a4f2881372","Type":"ContainerStarted","Data":"5f7effe10a28d1872a54d2caa2e046e535516b1d87b210c5c60c8091f4375dfd"} Jul 21 21:54:05.970070 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.969993 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:05.983002 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.982956 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" podStartSLOduration=2.467457661 podStartE2EDuration="7.982939359s" totalImagesPullingTime="5.515481698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:05.982050094 +0000 UTC m=+346.088014241" watchObservedRunningTime="2026-07-21 21:54:05.982939359 +0000 UTC m=+346.088903504" Jul 21 21:54:05.998718 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:05.998658 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" podStartSLOduration=2.670589159 podStartE2EDuration="6.998638308s" totalImagesPullingTime="4.328049149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:05.99798574 +0000 UTC m=+346.103949884" watchObservedRunningTime="2026-07-21 21:54:05.998638308 +0000 UTC m=+346.104602454" Jul 21 21:54:06.018184 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:06.017471 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-594787b58-x8sdj" podStartSLOduration=2.483397905 podStartE2EDuration="7.017434564s" totalImagesPullingTime="4.534036659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:53:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:06.015520595 +0000 UTC m=+346.121484742" watchObservedRunningTime="2026-07-21 21:54:06.017434564 +0000 UTC m=+346.123398714" Jul 21 21:54:06.038778 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:06.038720 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" podStartSLOduration=1.849720112 podStartE2EDuration="6.038701803s" totalImagesPullingTime="4.188981691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:06.036831143 +0000 UTC m=+346.142795311" watchObservedRunningTime="2026-07-21 21:54:06.038701803 +0000 UTC m=+346.144665950" Jul 21 21:54:06.977503 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:06.977474 2557 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-csc4j container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 21 21:54:06.977891 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:06.977528 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" podUID="f381058e-1c7f-4e93-a9c0-3b875f0500c9" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 21 21:54:08.625143 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.625085 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk"] Jul 21 21:54:08.628753 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.628728 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.631350 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.631313 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 21 21:54:08.632042 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.632025 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 21 21:54:08.632157 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.632046 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 21 21:54:08.632222 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.632150 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 21 21:54:08.632292 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.632272 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-7jjg6\"" Jul 21 21:54:08.632339 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.632312 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 21 21:54:08.640717 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.640693 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk"] Jul 21 21:54:08.662118 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.662073 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-884xl\" (UniqueName: \"kubernetes.io/projected/17f04541-80bc-48d5-9d51-7e7315092316-kube-api-access-884xl\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.662277 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.662146 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/17f04541-80bc-48d5-9d51-7e7315092316-manager-config\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.662277 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.662205 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17f04541-80bc-48d5-9d51-7e7315092316-metrics-cert\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.662277 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.662248 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f04541-80bc-48d5-9d51-7e7315092316-cert\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.762870 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.762834 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-884xl\" (UniqueName: \"kubernetes.io/projected/17f04541-80bc-48d5-9d51-7e7315092316-kube-api-access-884xl\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.763068 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.762884 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/17f04541-80bc-48d5-9d51-7e7315092316-manager-config\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.763068 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.762925 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17f04541-80bc-48d5-9d51-7e7315092316-metrics-cert\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.763068 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.762942 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f04541-80bc-48d5-9d51-7e7315092316-cert\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.763688 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.763663 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/17f04541-80bc-48d5-9d51-7e7315092316-manager-config\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.765506 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.765485 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/17f04541-80bc-48d5-9d51-7e7315092316-metrics-cert\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.765596 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.765565 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/17f04541-80bc-48d5-9d51-7e7315092316-cert\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.781126 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.781083 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-884xl\" (UniqueName: \"kubernetes.io/projected/17f04541-80bc-48d5-9d51-7e7315092316-kube-api-access-884xl\") pod \"lws-controller-manager-d89c66ff-dvzhk\" (UID: \"17f04541-80bc-48d5-9d51-7e7315092316\") " pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:08.938798 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:08.938695 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:09.092505 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:09.092475 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk"] Jul 21 21:54:09.094373 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:54:09.094347 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f04541_80bc_48d5_9d51_7e7315092316.slice/crio-9ecba8b4f20959efc1550c6cc0b35a295ecf4e360be948fb9d7726dc744f6c53 WatchSource:0}: Error finding container 9ecba8b4f20959efc1550c6cc0b35a295ecf4e360be948fb9d7726dc744f6c53: Status 404 returned error can't find the container with id 9ecba8b4f20959efc1550c6cc0b35a295ecf4e360be948fb9d7726dc744f6c53 Jul 21 21:54:09.977027 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:09.976997 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:09.992900 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:09.992863 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" event={"ID":"17f04541-80bc-48d5-9d51-7e7315092316","Type":"ContainerStarted","Data":"9ecba8b4f20959efc1550c6cc0b35a295ecf4e360be948fb9d7726dc744f6c53"} Jul 21 21:54:11.000198 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:11.000152 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" event={"ID":"17f04541-80bc-48d5-9d51-7e7315092316","Type":"ContainerStarted","Data":"ce16c06be28be5236c6719bf02e6014dcf64d1772d9c63d92f9f629567949b0a"} Jul 21 21:54:11.000671 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:11.000242 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:54:11.028201 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:11.028144 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" podStartSLOduration=1.287061649 podStartE2EDuration="3.028127814s" totalImagesPullingTime="1.741066165s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:54:11.025762422 +0000 UTC m=+351.131726568" watchObservedRunningTime="2026-07-21 21:54:11.028127814 +0000 UTC m=+351.134092038" Jul 21 21:54:13.063422 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.060930 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-csc4j"] Jul 21 21:54:13.063422 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.061185 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" podUID="f381058e-1c7f-4e93-a9c0-3b875f0500c9" containerName="discovery" containerID="cri-o://62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4" gracePeriod=30 Jul 21 21:54:13.317307 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.317251 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:13.403408 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.403381 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f381058e-1c7f-4e93-a9c0-3b875f0500c9-local-certs\") pod \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " Jul 21 21:54:13.403616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.403433 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-cacerts\") pod \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " Jul 21 21:54:13.403616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.403469 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-dns-cert\") pod \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " Jul 21 21:54:13.403616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.403492 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-ca-configmap\") pod \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " Jul 21 21:54:13.403616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.403531 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-kubeconfig\") pod \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " Jul 21 21:54:13.403616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.403559 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-token\") pod \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " Jul 21 21:54:13.403616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.403584 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4ldn\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-kube-api-access-z4ldn\") pod \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\" (UID: \"f381058e-1c7f-4e93-a9c0-3b875f0500c9\") " Jul 21 21:54:13.403964 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.403938 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "f381058e-1c7f-4e93-a9c0-3b875f0500c9" (UID: "f381058e-1c7f-4e93-a9c0-3b875f0500c9"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:54:13.406152 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.406123 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "f381058e-1c7f-4e93-a9c0-3b875f0500c9" (UID: "f381058e-1c7f-4e93-a9c0-3b875f0500c9"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:54:13.406152 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.406130 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-kube-api-access-z4ldn" (OuterVolumeSpecName: "kube-api-access-z4ldn") pod "f381058e-1c7f-4e93-a9c0-3b875f0500c9" (UID: "f381058e-1c7f-4e93-a9c0-3b875f0500c9"). InnerVolumeSpecName "kube-api-access-z4ldn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:54:13.406326 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.406235 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f381058e-1c7f-4e93-a9c0-3b875f0500c9-local-certs" (OuterVolumeSpecName: "local-certs") pod "f381058e-1c7f-4e93-a9c0-3b875f0500c9" (UID: "f381058e-1c7f-4e93-a9c0-3b875f0500c9"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:54:13.406376 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.406334 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "f381058e-1c7f-4e93-a9c0-3b875f0500c9" (UID: "f381058e-1c7f-4e93-a9c0-3b875f0500c9"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:54:13.406424 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.406408 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-cacerts" (OuterVolumeSpecName: "cacerts") pod "f381058e-1c7f-4e93-a9c0-3b875f0500c9" (UID: "f381058e-1c7f-4e93-a9c0-3b875f0500c9"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:54:13.406497 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.406484 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-token" (OuterVolumeSpecName: "istio-token") pod "f381058e-1c7f-4e93-a9c0-3b875f0500c9" (UID: "f381058e-1c7f-4e93-a9c0-3b875f0500c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:54:13.504405 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.504374 2557 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-token\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:54:13.504405 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.504401 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4ldn\" (UniqueName: \"kubernetes.io/projected/f381058e-1c7f-4e93-a9c0-3b875f0500c9-kube-api-access-z4ldn\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:54:13.504405 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.504413 2557 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/f381058e-1c7f-4e93-a9c0-3b875f0500c9-local-certs\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:54:13.504648 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.504422 2557 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-cacerts\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:54:13.504648 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.504430 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-dns-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:54:13.504648 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.504440 2557 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-csr-ca-configmap\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:54:13.504648 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:13.504448 2557 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f381058e-1c7f-4e93-a9c0-3b875f0500c9-istio-kubeconfig\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:54:14.017877 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.017839 2557 generic.go:356] "Generic (PLEG): container finished" podID="f381058e-1c7f-4e93-a9c0-3b875f0500c9" containerID="62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4" exitCode=0 Jul 21 21:54:14.018142 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.017889 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" Jul 21 21:54:14.018142 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.017912 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" event={"ID":"f381058e-1c7f-4e93-a9c0-3b875f0500c9","Type":"ContainerDied","Data":"62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4"} Jul 21 21:54:14.018142 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.017952 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-csc4j" event={"ID":"f381058e-1c7f-4e93-a9c0-3b875f0500c9","Type":"ContainerDied","Data":"897201b44c49d1be561a6aff7d680bc53b419efd950bd3cfb67d07843bfcb3e4"} Jul 21 21:54:14.018142 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.017972 2557 scope.go:117] "RemoveContainer" containerID="62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4" Jul 21 21:54:14.026981 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.026963 2557 scope.go:117] "RemoveContainer" containerID="62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4" Jul 21 21:54:14.027266 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:54:14.027248 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4\": container with ID starting with 62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4 not found: ID does not exist" containerID="62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4" Jul 21 21:54:14.027314 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.027276 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4"} err="failed to get container status \"62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4\": rpc error: code = NotFound desc = could not find container \"62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4\": container with ID starting with 62eb591990267d4de221f3810edebc8f24e330d2b9e78d674bcf9919d43433d4 not found: ID does not exist" Jul 21 21:54:14.045696 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.045664 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-csc4j"] Jul 21 21:54:14.052384 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.052358 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-csc4j"] Jul 21 21:54:14.393937 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:14.393854 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f381058e-1c7f-4e93-a9c0-3b875f0500c9" path="/var/lib/kubelet/pods/f381058e-1c7f-4e93-a9c0-3b875f0500c9/volumes" Jul 21 21:54:16.977868 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:16.977828 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-n4qw9" Jul 21 21:54:16.981014 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:16.980978 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zr6gw" Jul 21 21:54:22.008894 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:54:22.008858 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-d89c66ff-dvzhk" Jul 21 21:55:14.984054 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.984014 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245"] Jul 21 21:55:14.984624 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.984484 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f381058e-1c7f-4e93-a9c0-3b875f0500c9" containerName="discovery" Jul 21 21:55:14.984624 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.984504 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f381058e-1c7f-4e93-a9c0-3b875f0500c9" containerName="discovery" Jul 21 21:55:14.984624 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.984615 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f381058e-1c7f-4e93-a9c0-3b875f0500c9" containerName="discovery" Jul 21 21:55:14.987798 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.987770 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:14.990519 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.990491 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 21:55:14.991356 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.991339 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cpkrr\"" Jul 21 21:55:14.991430 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.991343 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 21:55:15.007665 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:14.995960 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245"] Jul 21 21:55:15.181351 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.181301 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.181546 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.181375 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.181546 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.181399 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5svq\" (UniqueName: \"kubernetes.io/projected/5c647702-75d8-419b-9d6b-2b6ba74f5080-kube-api-access-q5svq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.282384 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.282280 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.282384 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.282368 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.282618 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.282398 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5svq\" (UniqueName: \"kubernetes.io/projected/5c647702-75d8-419b-9d6b-2b6ba74f5080-kube-api-access-q5svq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.282759 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.282738 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.282831 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.282785 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.291673 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.291631 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5svq\" (UniqueName: \"kubernetes.io/projected/5c647702-75d8-419b-9d6b-2b6ba74f5080-kube-api-access-q5svq\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.301362 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.301337 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:15.446971 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.446947 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245"] Jul 21 21:55:15.448923 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:55:15.448895 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c647702_75d8_419b_9d6b_2b6ba74f5080.slice/crio-ac90918d400e2af7aacfa700f9f134834e7c0103d280a6f000911022f142afbc WatchSource:0}: Error finding container ac90918d400e2af7aacfa700f9f134834e7c0103d280a6f000911022f142afbc: Status 404 returned error can't find the container with id ac90918d400e2af7aacfa700f9f134834e7c0103d280a6f000911022f142afbc Jul 21 21:55:15.587856 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.587821 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7"] Jul 21 21:55:15.591382 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.591358 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.599368 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.599338 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7"] Jul 21 21:55:15.787110 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.787051 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qdvl\" (UniqueName: \"kubernetes.io/projected/54dda3dd-b498-450e-880d-89c685642624-kube-api-access-7qdvl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.787110 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.787112 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.787369 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.787230 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.888467 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.888373 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.888467 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.888443 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qdvl\" (UniqueName: \"kubernetes.io/projected/54dda3dd-b498-450e-880d-89c685642624-kube-api-access-7qdvl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.888467 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.888469 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.888819 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.888798 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.888887 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.888820 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.897839 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.897800 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qdvl\" (UniqueName: \"kubernetes.io/projected/54dda3dd-b498-450e-880d-89c685642624-kube-api-access-7qdvl\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:15.929322 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:15.929279 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:16.069815 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.069788 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7"] Jul 21 21:55:16.072517 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:55:16.072485 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dda3dd_b498_450e_880d_89c685642624.slice/crio-f4a82b9451d922dcf52ce0fd563882b0a5f61f44a4143c53b08864cc7498347b WatchSource:0}: Error finding container f4a82b9451d922dcf52ce0fd563882b0a5f61f44a4143c53b08864cc7498347b: Status 404 returned error can't find the container with id f4a82b9451d922dcf52ce0fd563882b0a5f61f44a4143c53b08864cc7498347b Jul 21 21:55:16.187872 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.187838 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k"] Jul 21 21:55:16.191400 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.191375 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.191779 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.191753 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.191871 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.191822 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrbxz\" (UniqueName: \"kubernetes.io/projected/10c555e8-5f68-4a6c-bae0-4223d26350d7-kube-api-access-lrbxz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.191936 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.191878 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.200294 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.200268 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k"] Jul 21 21:55:16.293016 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.292977 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.293244 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.293041 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrbxz\" (UniqueName: \"kubernetes.io/projected/10c555e8-5f68-4a6c-bae0-4223d26350d7-kube-api-access-lrbxz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.293244 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.293114 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.293479 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.293457 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.293479 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.293473 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.302442 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.302414 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrbxz\" (UniqueName: \"kubernetes.io/projected/10c555e8-5f68-4a6c-bae0-4223d26350d7-kube-api-access-lrbxz\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.309933 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.309899 2557 generic.go:356] "Generic (PLEG): container finished" podID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerID="712d54c3ec3b24a1a2d15d89db350cb608794043c2a0004535d7b55cb5cf60b3" exitCode=0 Jul 21 21:55:16.310109 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.309982 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" event={"ID":"5c647702-75d8-419b-9d6b-2b6ba74f5080","Type":"ContainerDied","Data":"712d54c3ec3b24a1a2d15d89db350cb608794043c2a0004535d7b55cb5cf60b3"} Jul 21 21:55:16.310109 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.310038 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" event={"ID":"5c647702-75d8-419b-9d6b-2b6ba74f5080","Type":"ContainerStarted","Data":"ac90918d400e2af7aacfa700f9f134834e7c0103d280a6f000911022f142afbc"} Jul 21 21:55:16.311840 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.311815 2557 generic.go:356] "Generic (PLEG): container finished" podID="54dda3dd-b498-450e-880d-89c685642624" containerID="fba39eeaf54d9546ae8adaad35591f70e84f6dff248918393f30460b1ce63574" exitCode=0 Jul 21 21:55:16.311937 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.311884 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" event={"ID":"54dda3dd-b498-450e-880d-89c685642624","Type":"ContainerDied","Data":"fba39eeaf54d9546ae8adaad35591f70e84f6dff248918393f30460b1ce63574"} Jul 21 21:55:16.311937 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.311932 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" event={"ID":"54dda3dd-b498-450e-880d-89c685642624","Type":"ContainerStarted","Data":"f4a82b9451d922dcf52ce0fd563882b0a5f61f44a4143c53b08864cc7498347b"} Jul 21 21:55:16.540830 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.540796 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:16.587055 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.587015 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p"] Jul 21 21:55:16.590969 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.590950 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.602609 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.600403 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p"] Jul 21 21:55:16.602609 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.600966 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.602609 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.601016 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.602609 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.601045 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7tmd\" (UniqueName: \"kubernetes.io/projected/3da4e59f-0739-4ae2-a15d-8f08843a165e-kube-api-access-f7tmd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.687729 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.687693 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k"] Jul 21 21:55:16.691239 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:55:16.691210 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c555e8_5f68_4a6c_bae0_4223d26350d7.slice/crio-eda444df55ef81d3db6bceba5ad50d1c52cf7e12c43c8281e89bd4813a5d6d28 WatchSource:0}: Error finding container eda444df55ef81d3db6bceba5ad50d1c52cf7e12c43c8281e89bd4813a5d6d28: Status 404 returned error can't find the container with id eda444df55ef81d3db6bceba5ad50d1c52cf7e12c43c8281e89bd4813a5d6d28 Jul 21 21:55:16.702067 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.702043 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.702217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.702084 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.702217 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.702139 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7tmd\" (UniqueName: \"kubernetes.io/projected/3da4e59f-0739-4ae2-a15d-8f08843a165e-kube-api-access-f7tmd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.702420 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.702398 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.702506 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.702483 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.715431 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.715405 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7tmd\" (UniqueName: \"kubernetes.io/projected/3da4e59f-0739-4ae2-a15d-8f08843a165e-kube-api-access-f7tmd\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:16.907165 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:16.907123 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:17.089348 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.089303 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p"] Jul 21 21:55:17.099232 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:55:17.099184 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da4e59f_0739_4ae2_a15d_8f08843a165e.slice/crio-3e0dbbe0dfc0996f64d33bcdea3120f957280c499b48a6270e4e0111a380b1c3 WatchSource:0}: Error finding container 3e0dbbe0dfc0996f64d33bcdea3120f957280c499b48a6270e4e0111a380b1c3: Status 404 returned error can't find the container with id 3e0dbbe0dfc0996f64d33bcdea3120f957280c499b48a6270e4e0111a380b1c3 Jul 21 21:55:17.319110 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.319056 2557 generic.go:356] "Generic (PLEG): container finished" podID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerID="04a3d4f580c4a25af52465e7cc29a1f10923c71abb1c21a496b8b6c87f44ae0f" exitCode=0 Jul 21 21:55:17.319303 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.319129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" event={"ID":"3da4e59f-0739-4ae2-a15d-8f08843a165e","Type":"ContainerDied","Data":"04a3d4f580c4a25af52465e7cc29a1f10923c71abb1c21a496b8b6c87f44ae0f"} Jul 21 21:55:17.319303 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.319191 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" event={"ID":"3da4e59f-0739-4ae2-a15d-8f08843a165e","Type":"ContainerStarted","Data":"3e0dbbe0dfc0996f64d33bcdea3120f957280c499b48a6270e4e0111a380b1c3"} Jul 21 21:55:17.321209 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.321182 2557 generic.go:356] "Generic (PLEG): container finished" podID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerID="666061d9c0df6f0af20559c7b3a58a9933ce9831697ab943d4e885c43a1eb913" exitCode=0 Jul 21 21:55:17.321339 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.321264 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" event={"ID":"5c647702-75d8-419b-9d6b-2b6ba74f5080","Type":"ContainerDied","Data":"666061d9c0df6f0af20559c7b3a58a9933ce9831697ab943d4e885c43a1eb913"} Jul 21 21:55:17.323524 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.323503 2557 generic.go:356] "Generic (PLEG): container finished" podID="54dda3dd-b498-450e-880d-89c685642624" containerID="213077a7706c6e205980e29659f349feb36f40bcdcb779ec4bc7b0d39350af1a" exitCode=0 Jul 21 21:55:17.323632 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.323592 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" event={"ID":"54dda3dd-b498-450e-880d-89c685642624","Type":"ContainerDied","Data":"213077a7706c6e205980e29659f349feb36f40bcdcb779ec4bc7b0d39350af1a"} Jul 21 21:55:17.325139 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.325119 2557 generic.go:356] "Generic (PLEG): container finished" podID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerID="f8fcaab9a88b70f8826c3b926171f2998ea7c336ec6a8ce7055e5893be3567b4" exitCode=0 Jul 21 21:55:17.325233 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.325187 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" event={"ID":"10c555e8-5f68-4a6c-bae0-4223d26350d7","Type":"ContainerDied","Data":"f8fcaab9a88b70f8826c3b926171f2998ea7c336ec6a8ce7055e5893be3567b4"} Jul 21 21:55:17.325233 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:17.325223 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" event={"ID":"10c555e8-5f68-4a6c-bae0-4223d26350d7","Type":"ContainerStarted","Data":"eda444df55ef81d3db6bceba5ad50d1c52cf7e12c43c8281e89bd4813a5d6d28"} Jul 21 21:55:18.331833 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:18.331803 2557 generic.go:356] "Generic (PLEG): container finished" podID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerID="a062ece8e15b82d6b2d4a579fd4a3e55a6ee21bf9f9d98c607305e7de50b098e" exitCode=0 Jul 21 21:55:18.332368 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:18.331896 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" event={"ID":"3da4e59f-0739-4ae2-a15d-8f08843a165e","Type":"ContainerDied","Data":"a062ece8e15b82d6b2d4a579fd4a3e55a6ee21bf9f9d98c607305e7de50b098e"} Jul 21 21:55:18.334155 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:18.334127 2557 generic.go:356] "Generic (PLEG): container finished" podID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerID="a4bad0accf9c9a37395d4a7287db348ece985bad53379093c1853d6e9032d0de" exitCode=0 Jul 21 21:55:18.334251 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:18.334169 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" event={"ID":"5c647702-75d8-419b-9d6b-2b6ba74f5080","Type":"ContainerDied","Data":"a4bad0accf9c9a37395d4a7287db348ece985bad53379093c1853d6e9032d0de"} Jul 21 21:55:18.336305 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:18.336283 2557 generic.go:356] "Generic (PLEG): container finished" podID="54dda3dd-b498-450e-880d-89c685642624" containerID="16e5cda31ce14e51218b401604d4f0ca6a9f3a7f5963c57df5e85f8ec3831b63" exitCode=0 Jul 21 21:55:18.336420 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:18.336363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" event={"ID":"54dda3dd-b498-450e-880d-89c685642624","Type":"ContainerDied","Data":"16e5cda31ce14e51218b401604d4f0ca6a9f3a7f5963c57df5e85f8ec3831b63"} Jul 21 21:55:18.338373 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:18.338349 2557 generic.go:356] "Generic (PLEG): container finished" podID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerID="a3515b154201dcf3cee3bebebfe228ea95310859bd565bf67fee14ebcc947879" exitCode=0 Jul 21 21:55:18.338497 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:18.338371 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" event={"ID":"10c555e8-5f68-4a6c-bae0-4223d26350d7","Type":"ContainerDied","Data":"a3515b154201dcf3cee3bebebfe228ea95310859bd565bf67fee14ebcc947879"} Jul 21 21:55:19.345419 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.345383 2557 generic.go:356] "Generic (PLEG): container finished" podID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerID="9f2ed50424e8e2482fe8d72cfc485c60dec274f28a7a5773e958c3e89a6651ad" exitCode=0 Jul 21 21:55:19.345820 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.345461 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" event={"ID":"3da4e59f-0739-4ae2-a15d-8f08843a165e","Type":"ContainerDied","Data":"9f2ed50424e8e2482fe8d72cfc485c60dec274f28a7a5773e958c3e89a6651ad"} Jul 21 21:55:19.347502 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.347475 2557 generic.go:356] "Generic (PLEG): container finished" podID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerID="05f844fadec21544d0627eb37852a4ac15234b353e65e00089c21daab56c5509" exitCode=0 Jul 21 21:55:19.347616 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.347560 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" event={"ID":"10c555e8-5f68-4a6c-bae0-4223d26350d7","Type":"ContainerDied","Data":"05f844fadec21544d0627eb37852a4ac15234b353e65e00089c21daab56c5509"} Jul 21 21:55:19.482019 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.481989 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:19.520837 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.520807 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:19.521043 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.521024 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-util\") pod \"5c647702-75d8-419b-9d6b-2b6ba74f5080\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " Jul 21 21:55:19.521126 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.521066 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q5svq\" (UniqueName: \"kubernetes.io/projected/5c647702-75d8-419b-9d6b-2b6ba74f5080-kube-api-access-q5svq\") pod \"5c647702-75d8-419b-9d6b-2b6ba74f5080\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " Jul 21 21:55:19.521126 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.521116 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-bundle\") pod \"5c647702-75d8-419b-9d6b-2b6ba74f5080\" (UID: \"5c647702-75d8-419b-9d6b-2b6ba74f5080\") " Jul 21 21:55:19.521663 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.521637 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-bundle" (OuterVolumeSpecName: "bundle") pod "5c647702-75d8-419b-9d6b-2b6ba74f5080" (UID: "5c647702-75d8-419b-9d6b-2b6ba74f5080"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:19.523571 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.523536 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c647702-75d8-419b-9d6b-2b6ba74f5080-kube-api-access-q5svq" (OuterVolumeSpecName: "kube-api-access-q5svq") pod "5c647702-75d8-419b-9d6b-2b6ba74f5080" (UID: "5c647702-75d8-419b-9d6b-2b6ba74f5080"). InnerVolumeSpecName "kube-api-access-q5svq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:55:19.526327 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.526303 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-util" (OuterVolumeSpecName: "util") pod "5c647702-75d8-419b-9d6b-2b6ba74f5080" (UID: "5c647702-75d8-419b-9d6b-2b6ba74f5080"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:19.622435 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.622330 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-bundle\") pod \"54dda3dd-b498-450e-880d-89c685642624\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " Jul 21 21:55:19.622435 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.622412 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7qdvl\" (UniqueName: \"kubernetes.io/projected/54dda3dd-b498-450e-880d-89c685642624-kube-api-access-7qdvl\") pod \"54dda3dd-b498-450e-880d-89c685642624\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " Jul 21 21:55:19.622634 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.622449 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-util\") pod \"54dda3dd-b498-450e-880d-89c685642624\" (UID: \"54dda3dd-b498-450e-880d-89c685642624\") " Jul 21 21:55:19.622683 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.622633 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-util\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:19.622683 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.622651 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5svq\" (UniqueName: \"kubernetes.io/projected/5c647702-75d8-419b-9d6b-2b6ba74f5080-kube-api-access-q5svq\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:19.622683 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.622666 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5c647702-75d8-419b-9d6b-2b6ba74f5080-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:19.622870 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.622844 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-bundle" (OuterVolumeSpecName: "bundle") pod "54dda3dd-b498-450e-880d-89c685642624" (UID: "54dda3dd-b498-450e-880d-89c685642624"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:19.624751 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.624721 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54dda3dd-b498-450e-880d-89c685642624-kube-api-access-7qdvl" (OuterVolumeSpecName: "kube-api-access-7qdvl") pod "54dda3dd-b498-450e-880d-89c685642624" (UID: "54dda3dd-b498-450e-880d-89c685642624"). InnerVolumeSpecName "kube-api-access-7qdvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:55:19.628064 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.628035 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-util" (OuterVolumeSpecName: "util") pod "54dda3dd-b498-450e-880d-89c685642624" (UID: "54dda3dd-b498-450e-880d-89c685642624"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:19.723707 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.723643 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:19.723707 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.723695 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7qdvl\" (UniqueName: \"kubernetes.io/projected/54dda3dd-b498-450e-880d-89c685642624-kube-api-access-7qdvl\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:19.723707 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:19.723708 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/54dda3dd-b498-450e-880d-89c685642624-util\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:20.353975 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.353940 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" Jul 21 21:55:20.353975 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.353964 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245" event={"ID":"5c647702-75d8-419b-9d6b-2b6ba74f5080","Type":"ContainerDied","Data":"ac90918d400e2af7aacfa700f9f134834e7c0103d280a6f000911022f142afbc"} Jul 21 21:55:20.354457 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.354045 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac90918d400e2af7aacfa700f9f134834e7c0103d280a6f000911022f142afbc" Jul 21 21:55:20.355905 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.355880 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" Jul 21 21:55:20.356037 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.355905 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7" event={"ID":"54dda3dd-b498-450e-880d-89c685642624","Type":"ContainerDied","Data":"f4a82b9451d922dcf52ce0fd563882b0a5f61f44a4143c53b08864cc7498347b"} Jul 21 21:55:20.356037 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.355944 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f4a82b9451d922dcf52ce0fd563882b0a5f61f44a4143c53b08864cc7498347b" Jul 21 21:55:20.501732 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.501696 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:20.530238 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.530207 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-bundle\") pod \"10c555e8-5f68-4a6c-bae0-4223d26350d7\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " Jul 21 21:55:20.530436 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.530293 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lrbxz\" (UniqueName: \"kubernetes.io/projected/10c555e8-5f68-4a6c-bae0-4223d26350d7-kube-api-access-lrbxz\") pod \"10c555e8-5f68-4a6c-bae0-4223d26350d7\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " Jul 21 21:55:20.530436 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.530388 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-util\") pod \"10c555e8-5f68-4a6c-bae0-4223d26350d7\" (UID: \"10c555e8-5f68-4a6c-bae0-4223d26350d7\") " Jul 21 21:55:20.530820 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.530791 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-bundle" (OuterVolumeSpecName: "bundle") pod "10c555e8-5f68-4a6c-bae0-4223d26350d7" (UID: "10c555e8-5f68-4a6c-bae0-4223d26350d7"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:20.533392 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.533366 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c555e8-5f68-4a6c-bae0-4223d26350d7-kube-api-access-lrbxz" (OuterVolumeSpecName: "kube-api-access-lrbxz") pod "10c555e8-5f68-4a6c-bae0-4223d26350d7" (UID: "10c555e8-5f68-4a6c-bae0-4223d26350d7"). InnerVolumeSpecName "kube-api-access-lrbxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:55:20.535193 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.535174 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:20.536930 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.536906 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-util" (OuterVolumeSpecName: "util") pod "10c555e8-5f68-4a6c-bae0-4223d26350d7" (UID: "10c555e8-5f68-4a6c-bae0-4223d26350d7"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:20.631341 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.631238 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-util\") pod \"3da4e59f-0739-4ae2-a15d-8f08843a165e\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " Jul 21 21:55:20.631341 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.631293 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7tmd\" (UniqueName: \"kubernetes.io/projected/3da4e59f-0739-4ae2-a15d-8f08843a165e-kube-api-access-f7tmd\") pod \"3da4e59f-0739-4ae2-a15d-8f08843a165e\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " Jul 21 21:55:20.631595 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.631385 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-bundle\") pod \"3da4e59f-0739-4ae2-a15d-8f08843a165e\" (UID: \"3da4e59f-0739-4ae2-a15d-8f08843a165e\") " Jul 21 21:55:20.631649 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.631602 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrbxz\" (UniqueName: \"kubernetes.io/projected/10c555e8-5f68-4a6c-bae0-4223d26350d7-kube-api-access-lrbxz\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:20.631649 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.631619 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-util\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:20.631649 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.631632 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/10c555e8-5f68-4a6c-bae0-4223d26350d7-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:20.632004 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.631969 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-bundle" (OuterVolumeSpecName: "bundle") pod "3da4e59f-0739-4ae2-a15d-8f08843a165e" (UID: "3da4e59f-0739-4ae2-a15d-8f08843a165e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:20.633627 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.633602 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da4e59f-0739-4ae2-a15d-8f08843a165e-kube-api-access-f7tmd" (OuterVolumeSpecName: "kube-api-access-f7tmd") pod "3da4e59f-0739-4ae2-a15d-8f08843a165e" (UID: "3da4e59f-0739-4ae2-a15d-8f08843a165e"). InnerVolumeSpecName "kube-api-access-f7tmd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:55:20.636594 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.636566 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-util" (OuterVolumeSpecName: "util") pod "3da4e59f-0739-4ae2-a15d-8f08843a165e" (UID: "3da4e59f-0739-4ae2-a15d-8f08843a165e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 21:55:20.733036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.732991 2557 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:20.733036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.733025 2557 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/3da4e59f-0739-4ae2-a15d-8f08843a165e-util\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:20.733036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:20.733034 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7tmd\" (UniqueName: \"kubernetes.io/projected/3da4e59f-0739-4ae2-a15d-8f08843a165e-kube-api-access-f7tmd\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:55:21.362747 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:21.362715 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" Jul 21 21:55:21.362747 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:21.362733 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k" event={"ID":"10c555e8-5f68-4a6c-bae0-4223d26350d7","Type":"ContainerDied","Data":"eda444df55ef81d3db6bceba5ad50d1c52cf7e12c43c8281e89bd4813a5d6d28"} Jul 21 21:55:21.363237 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:21.362780 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eda444df55ef81d3db6bceba5ad50d1c52cf7e12c43c8281e89bd4813a5d6d28" Jul 21 21:55:21.364788 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:21.364759 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" event={"ID":"3da4e59f-0739-4ae2-a15d-8f08843a165e","Type":"ContainerDied","Data":"3e0dbbe0dfc0996f64d33bcdea3120f957280c499b48a6270e4e0111a380b1c3"} Jul 21 21:55:21.364897 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:21.364809 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e0dbbe0dfc0996f64d33bcdea3120f957280c499b48a6270e4e0111a380b1c3" Jul 21 21:55:21.364897 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:21.364766 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p" Jul 21 21:55:31.829070 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:31.829032 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbc76f785-vv87m"] Jul 21 21:55:42.821373 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821289 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd"] Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821646 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dda3dd-b498-450e-880d-89c685642624" containerName="util" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821659 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dda3dd-b498-450e-880d-89c685642624" containerName="util" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821671 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerName="extract" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821677 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerName="extract" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821686 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerName="extract" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821691 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerName="extract" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821699 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerName="util" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821704 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerName="util" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821709 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerName="pull" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821714 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerName="pull" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821722 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dda3dd-b498-450e-880d-89c685642624" containerName="pull" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821726 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dda3dd-b498-450e-880d-89c685642624" containerName="pull" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821735 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerName="extract" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821740 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerName="extract" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821752 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerName="pull" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821757 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerName="pull" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821763 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="54dda3dd-b498-450e-880d-89c685642624" containerName="extract" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821768 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="54dda3dd-b498-450e-880d-89c685642624" containerName="extract" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821774 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerName="util" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821779 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerName="util" Jul 21 21:55:42.821773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821785 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerName="pull" Jul 21 21:55:42.822460 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821789 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerName="pull" Jul 21 21:55:42.822460 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821796 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerName="util" Jul 21 21:55:42.822460 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821801 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerName="util" Jul 21 21:55:42.822460 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821853 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="54dda3dd-b498-450e-880d-89c685642624" containerName="extract" Jul 21 21:55:42.822460 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821861 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="3da4e59f-0739-4ae2-a15d-8f08843a165e" containerName="extract" Jul 21 21:55:42.822460 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821868 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c647702-75d8-419b-9d6b-2b6ba74f5080" containerName="extract" Jul 21 21:55:42.822460 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.821873 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="10c555e8-5f68-4a6c-bae0-4223d26350d7" containerName="extract" Jul 21 21:55:42.824679 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.824663 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:42.826818 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.826796 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 21 21:55:42.827326 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.827302 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 21 21:55:42.827434 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.827340 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 21 21:55:42.827834 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.827812 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cpkrr\"" Jul 21 21:55:42.827834 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.827828 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 21 21:55:42.833291 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.833239 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd"] Jul 21 21:55:42.928512 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.928472 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7bcd1e-4217-4651-8dfb-c33468138baa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:42.928687 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.928518 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cf7bcd1e-4217-4651-8dfb-c33468138baa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:42.928687 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:42.928561 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfxc\" (UniqueName: \"kubernetes.io/projected/cf7bcd1e-4217-4651-8dfb-c33468138baa-kube-api-access-pkfxc\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.029786 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.029737 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7bcd1e-4217-4651-8dfb-c33468138baa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.029786 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.029793 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cf7bcd1e-4217-4651-8dfb-c33468138baa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.029997 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.029830 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfxc\" (UniqueName: \"kubernetes.io/projected/cf7bcd1e-4217-4651-8dfb-c33468138baa-kube-api-access-pkfxc\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.029997 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:55:43.029903 2557 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 21 21:55:43.029997 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:55:43.029995 2557 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf7bcd1e-4217-4651-8dfb-c33468138baa-plugin-serving-cert podName:cf7bcd1e-4217-4651-8dfb-c33468138baa nodeName:}" failed. No retries permitted until 2026-07-21 21:55:43.529973307 +0000 UTC m=+443.635937451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/cf7bcd1e-4217-4651-8dfb-c33468138baa-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-jr5pd" (UID: "cf7bcd1e-4217-4651-8dfb-c33468138baa") : secret "plugin-serving-cert" not found Jul 21 21:55:43.030549 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.030529 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/cf7bcd1e-4217-4651-8dfb-c33468138baa-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.039104 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.039071 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfxc\" (UniqueName: \"kubernetes.io/projected/cf7bcd1e-4217-4651-8dfb-c33468138baa-kube-api-access-pkfxc\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.535157 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.535112 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7bcd1e-4217-4651-8dfb-c33468138baa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.537828 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.537801 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/cf7bcd1e-4217-4651-8dfb-c33468138baa-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-jr5pd\" (UID: \"cf7bcd1e-4217-4651-8dfb-c33468138baa\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.736059 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.736016 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" Jul 21 21:55:43.876083 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:43.876055 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd"] Jul 21 21:55:43.878527 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:55:43.878499 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcf7bcd1e_4217_4651_8dfb_c33468138baa.slice/crio-b61a03a1d8659745f77f9d343a85b1bcd21f176d17de059ee32f6734eadd4264 WatchSource:0}: Error finding container b61a03a1d8659745f77f9d343a85b1bcd21f176d17de059ee32f6734eadd4264: Status 404 returned error can't find the container with id b61a03a1d8659745f77f9d343a85b1bcd21f176d17de059ee32f6734eadd4264 Jul 21 21:55:44.489872 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:44.489836 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" event={"ID":"cf7bcd1e-4217-4651-8dfb-c33468138baa","Type":"ContainerStarted","Data":"b61a03a1d8659745f77f9d343a85b1bcd21f176d17de059ee32f6734eadd4264"} Jul 21 21:55:56.851401 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:55:56.851352 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7fbc76f785-vv87m" podUID="f87a0744-f78c-4faa-af31-a92c52d9067f" containerName="console" containerID="cri-o://01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0" gracePeriod=15 Jul 21 21:56:00.274450 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:00.274405 2557 patch_prober.go:28] interesting pod/console-7fbc76f785-vv87m container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" start-of-body= Jul 21 21:56:00.274849 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:00.274461 2557 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7fbc76f785-vv87m" podUID="f87a0744-f78c-4faa-af31-a92c52d9067f" containerName="console" probeResult="failure" output="Get \"https://10.132.0.23:8443/health\": dial tcp 10.132.0.23:8443: connect: connection refused" Jul 21 21:56:06.976664 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:06.976618 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:56:07.156638 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.156541 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-console-config\") pod \"f87a0744-f78c-4faa-af31-a92c52d9067f\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " Jul 21 21:56:07.156638 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.156592 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-oauth-config\") pod \"f87a0744-f78c-4faa-af31-a92c52d9067f\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " Jul 21 21:56:07.156638 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.156623 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-oauth-serving-cert\") pod \"f87a0744-f78c-4faa-af31-a92c52d9067f\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " Jul 21 21:56:07.156919 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.156674 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-service-ca\") pod \"f87a0744-f78c-4faa-af31-a92c52d9067f\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " Jul 21 21:56:07.156919 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.156773 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrlff\" (UniqueName: \"kubernetes.io/projected/f87a0744-f78c-4faa-af31-a92c52d9067f-kube-api-access-nrlff\") pod \"f87a0744-f78c-4faa-af31-a92c52d9067f\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " Jul 21 21:56:07.156919 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.156796 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-trusted-ca-bundle\") pod \"f87a0744-f78c-4faa-af31-a92c52d9067f\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " Jul 21 21:56:07.157071 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.156916 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-serving-cert\") pod \"f87a0744-f78c-4faa-af31-a92c52d9067f\" (UID: \"f87a0744-f78c-4faa-af31-a92c52d9067f\") " Jul 21 21:56:07.157155 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.157087 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-console-config" (OuterVolumeSpecName: "console-config") pod "f87a0744-f78c-4faa-af31-a92c52d9067f" (UID: "f87a0744-f78c-4faa-af31-a92c52d9067f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:56:07.157262 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.157221 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-service-ca" (OuterVolumeSpecName: "service-ca") pod "f87a0744-f78c-4faa-af31-a92c52d9067f" (UID: "f87a0744-f78c-4faa-af31-a92c52d9067f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:56:07.157262 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.157252 2557 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-console-config\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:56:07.157262 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.157251 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f87a0744-f78c-4faa-af31-a92c52d9067f" (UID: "f87a0744-f78c-4faa-af31-a92c52d9067f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:56:07.157684 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.157458 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f87a0744-f78c-4faa-af31-a92c52d9067f" (UID: "f87a0744-f78c-4faa-af31-a92c52d9067f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 21 21:56:07.159117 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.159060 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f87a0744-f78c-4faa-af31-a92c52d9067f" (UID: "f87a0744-f78c-4faa-af31-a92c52d9067f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:56:07.159117 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.159081 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f87a0744-f78c-4faa-af31-a92c52d9067f" (UID: "f87a0744-f78c-4faa-af31-a92c52d9067f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 21:56:07.159257 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.159082 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f87a0744-f78c-4faa-af31-a92c52d9067f-kube-api-access-nrlff" (OuterVolumeSpecName: "kube-api-access-nrlff") pod "f87a0744-f78c-4faa-af31-a92c52d9067f" (UID: "f87a0744-f78c-4faa-af31-a92c52d9067f"). InnerVolumeSpecName "kube-api-access-nrlff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:56:07.257973 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.257930 2557 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-service-ca\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:56:07.257973 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.257969 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrlff\" (UniqueName: \"kubernetes.io/projected/f87a0744-f78c-4faa-af31-a92c52d9067f-kube-api-access-nrlff\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:56:07.257973 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.257980 2557 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-trusted-ca-bundle\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:56:07.258249 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.257989 2557 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-serving-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:56:07.258249 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.257998 2557 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f87a0744-f78c-4faa-af31-a92c52d9067f-console-oauth-config\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:56:07.258249 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.258007 2557 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f87a0744-f78c-4faa-af31-a92c52d9067f-oauth-serving-cert\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:56:07.628896 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.628862 2557 generic.go:356] "Generic (PLEG): container finished" podID="f87a0744-f78c-4faa-af31-a92c52d9067f" containerID="01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0" exitCode=0 Jul 21 21:56:07.629137 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.628913 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7fbc76f785-vv87m" Jul 21 21:56:07.629137 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.628951 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbc76f785-vv87m" event={"ID":"f87a0744-f78c-4faa-af31-a92c52d9067f","Type":"ContainerDied","Data":"01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0"} Jul 21 21:56:07.629137 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.629008 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7fbc76f785-vv87m" event={"ID":"f87a0744-f78c-4faa-af31-a92c52d9067f","Type":"ContainerDied","Data":"8eae546432bbd7eed8bf7a2b2be88b45b0acabe94c5d2938bf86dd099e499819"} Jul 21 21:56:07.629137 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.629034 2557 scope.go:117] "RemoveContainer" containerID="01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0" Jul 21 21:56:07.630870 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.630845 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" event={"ID":"cf7bcd1e-4217-4651-8dfb-c33468138baa","Type":"ContainerStarted","Data":"36d1daf99c1d3b6ce0295cc7baf1f4a3c74b197391c745395b4c2628053635e6"} Jul 21 21:56:07.638028 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.638007 2557 scope.go:117] "RemoveContainer" containerID="01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0" Jul 21 21:56:07.638413 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:56:07.638391 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0\": container with ID starting with 01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0 not found: ID does not exist" containerID="01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0" Jul 21 21:56:07.638469 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.638421 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0"} err="failed to get container status \"01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0\": rpc error: code = NotFound desc = could not find container \"01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0\": container with ID starting with 01e9c2e2543c03c5356d9555576f39e491b8c0a2293b70be47e1a193883f27f0 not found: ID does not exist" Jul 21 21:56:07.667858 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.667800 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-jr5pd" podStartSLOduration=2.648847214 podStartE2EDuration="25.667782561s" totalImagesPullingTime="23.018935347s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:56:07.666356567 +0000 UTC m=+467.772320712" watchObservedRunningTime="2026-07-21 21:56:07.667782561 +0000 UTC m=+467.773746706" Jul 21 21:56:07.694819 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.694786 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7fbc76f785-vv87m"] Jul 21 21:56:07.702539 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:07.702505 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7fbc76f785-vv87m"] Jul 21 21:56:08.390787 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:08.390742 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87a0744-f78c-4faa-af31-a92c52d9067f" path="/var/lib/kubelet/pods/f87a0744-f78c-4faa-af31-a92c52d9067f/volumes" Jul 21 21:56:37.592969 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.592926 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:56:37.593475 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.593270 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87a0744-f78c-4faa-af31-a92c52d9067f" containerName="console" Jul 21 21:56:37.593475 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.593281 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87a0744-f78c-4faa-af31-a92c52d9067f" containerName="console" Jul 21 21:56:37.593475 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.593344 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87a0744-f78c-4faa-af31-a92c52d9067f" containerName="console" Jul 21 21:56:37.664778 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.664741 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:56:37.664778 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.664784 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:56:37.664994 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.664881 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:37.667313 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.667291 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 21 21:56:37.814838 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.814801 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcfhj\" (UniqueName: \"kubernetes.io/projected/fada2524-786d-4b4f-a0e9-81fe443006ad-kube-api-access-tcfhj\") pod \"limitador-limitador-54d6d988d5-lvrlh\" (UID: \"fada2524-786d-4b4f-a0e9-81fe443006ad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:37.815027 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.814864 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fada2524-786d-4b4f-a0e9-81fe443006ad-config-file\") pod \"limitador-limitador-54d6d988d5-lvrlh\" (UID: \"fada2524-786d-4b4f-a0e9-81fe443006ad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:37.916138 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.916035 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcfhj\" (UniqueName: \"kubernetes.io/projected/fada2524-786d-4b4f-a0e9-81fe443006ad-kube-api-access-tcfhj\") pod \"limitador-limitador-54d6d988d5-lvrlh\" (UID: \"fada2524-786d-4b4f-a0e9-81fe443006ad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:37.916138 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.916079 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fada2524-786d-4b4f-a0e9-81fe443006ad-config-file\") pod \"limitador-limitador-54d6d988d5-lvrlh\" (UID: \"fada2524-786d-4b4f-a0e9-81fe443006ad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:37.916718 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.916698 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/fada2524-786d-4b4f-a0e9-81fe443006ad-config-file\") pod \"limitador-limitador-54d6d988d5-lvrlh\" (UID: \"fada2524-786d-4b4f-a0e9-81fe443006ad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:37.924708 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.924673 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcfhj\" (UniqueName: \"kubernetes.io/projected/fada2524-786d-4b4f-a0e9-81fe443006ad-kube-api-access-tcfhj\") pod \"limitador-limitador-54d6d988d5-lvrlh\" (UID: \"fada2524-786d-4b4f-a0e9-81fe443006ad\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:37.974744 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:37.974710 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:38.064085 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.064050 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-q2w2f"] Jul 21 21:56:38.119191 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.119157 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-q2w2f"] Jul 21 21:56:38.119339 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.119262 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-q2w2f" Jul 21 21:56:38.121720 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.121701 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5swz\"" Jul 21 21:56:38.126943 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.126923 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:56:38.129412 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:56:38.129381 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfada2524_786d_4b4f_a0e9_81fe443006ad.slice/crio-bf10f3dca2e11dbdf46545c55fd9d3649d9deba2605a92959c972157cb9db5b7 WatchSource:0}: Error finding container bf10f3dca2e11dbdf46545c55fd9d3649d9deba2605a92959c972157cb9db5b7: Status 404 returned error can't find the container with id bf10f3dca2e11dbdf46545c55fd9d3649d9deba2605a92959c972157cb9db5b7 Jul 21 21:56:38.219301 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.219204 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg6vl\" (UniqueName: \"kubernetes.io/projected/442d5096-f83c-46f8-b4f8-79a0096899be-kube-api-access-zg6vl\") pod \"authorino-58cd955675-q2w2f\" (UID: \"442d5096-f83c-46f8-b4f8-79a0096899be\") " pod="kuadrant-system/authorino-58cd955675-q2w2f" Jul 21 21:56:38.320534 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.320498 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg6vl\" (UniqueName: \"kubernetes.io/projected/442d5096-f83c-46f8-b4f8-79a0096899be-kube-api-access-zg6vl\") pod \"authorino-58cd955675-q2w2f\" (UID: \"442d5096-f83c-46f8-b4f8-79a0096899be\") " pod="kuadrant-system/authorino-58cd955675-q2w2f" Jul 21 21:56:38.330372 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.330343 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg6vl\" (UniqueName: \"kubernetes.io/projected/442d5096-f83c-46f8-b4f8-79a0096899be-kube-api-access-zg6vl\") pod \"authorino-58cd955675-q2w2f\" (UID: \"442d5096-f83c-46f8-b4f8-79a0096899be\") " pod="kuadrant-system/authorino-58cd955675-q2w2f" Jul 21 21:56:38.430168 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.430140 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-q2w2f" Jul 21 21:56:38.668748 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.668714 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-q2w2f"] Jul 21 21:56:38.670802 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:56:38.670775 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod442d5096_f83c_46f8_b4f8_79a0096899be.slice/crio-e0fce1a23b4e41e4a6368feed1d35080b45ae808f497a6c64dfa9790bf72d9a6 WatchSource:0}: Error finding container e0fce1a23b4e41e4a6368feed1d35080b45ae808f497a6c64dfa9790bf72d9a6: Status 404 returned error can't find the container with id e0fce1a23b4e41e4a6368feed1d35080b45ae808f497a6c64dfa9790bf72d9a6 Jul 21 21:56:38.788332 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.788212 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" event={"ID":"fada2524-786d-4b4f-a0e9-81fe443006ad","Type":"ContainerStarted","Data":"bf10f3dca2e11dbdf46545c55fd9d3649d9deba2605a92959c972157cb9db5b7"} Jul 21 21:56:38.789720 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:38.789686 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-q2w2f" event={"ID":"442d5096-f83c-46f8-b4f8-79a0096899be","Type":"ContainerStarted","Data":"e0fce1a23b4e41e4a6368feed1d35080b45ae808f497a6c64dfa9790bf72d9a6"} Jul 21 21:56:42.819967 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:42.819542 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-q2w2f" event={"ID":"442d5096-f83c-46f8-b4f8-79a0096899be","Type":"ContainerStarted","Data":"7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d"} Jul 21 21:56:42.822043 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:42.822017 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" event={"ID":"fada2524-786d-4b4f-a0e9-81fe443006ad","Type":"ContainerStarted","Data":"17a63ec01f664f29dd550bd2a102262997cca5d1783e07e2bac92e4245d00b69"} Jul 21 21:56:42.822190 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:42.822134 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:56:42.836924 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:42.836863 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-q2w2f" podStartSLOduration=0.811025405 podStartE2EDuration="4.836842597s" totalImagesPullingTime="4.025817192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:56:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:56:42.835549081 +0000 UTC m=+502.941513238" watchObservedRunningTime="2026-07-21 21:56:42.836842597 +0000 UTC m=+502.942806745" Jul 21 21:56:42.854448 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:42.854396 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" podStartSLOduration=1.293962018 podStartE2EDuration="5.854378227s" totalImagesPullingTime="4.560416209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:56:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:56:42.852224957 +0000 UTC m=+502.958189124" watchObservedRunningTime="2026-07-21 21:56:42.854378227 +0000 UTC m=+502.960342380" Jul 21 21:56:43.183673 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:43.183640 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-q2w2f"] Jul 21 21:56:44.832365 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:44.832328 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-q2w2f" podUID="442d5096-f83c-46f8-b4f8-79a0096899be" containerName="authorino" containerID="cri-o://7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d" gracePeriod=30 Jul 21 21:56:45.081946 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.081920 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-q2w2f" Jul 21 21:56:45.191036 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.190952 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg6vl\" (UniqueName: \"kubernetes.io/projected/442d5096-f83c-46f8-b4f8-79a0096899be-kube-api-access-zg6vl\") pod \"442d5096-f83c-46f8-b4f8-79a0096899be\" (UID: \"442d5096-f83c-46f8-b4f8-79a0096899be\") " Jul 21 21:56:45.193161 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.193131 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/442d5096-f83c-46f8-b4f8-79a0096899be-kube-api-access-zg6vl" (OuterVolumeSpecName: "kube-api-access-zg6vl") pod "442d5096-f83c-46f8-b4f8-79a0096899be" (UID: "442d5096-f83c-46f8-b4f8-79a0096899be"). InnerVolumeSpecName "kube-api-access-zg6vl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 21:56:45.292470 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.292428 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg6vl\" (UniqueName: \"kubernetes.io/projected/442d5096-f83c-46f8-b4f8-79a0096899be-kube-api-access-zg6vl\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 21:56:45.838577 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.838536 2557 generic.go:356] "Generic (PLEG): container finished" podID="442d5096-f83c-46f8-b4f8-79a0096899be" containerID="7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d" exitCode=0 Jul 21 21:56:45.838995 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.838592 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-q2w2f" Jul 21 21:56:45.838995 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.838619 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-q2w2f" event={"ID":"442d5096-f83c-46f8-b4f8-79a0096899be","Type":"ContainerDied","Data":"7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d"} Jul 21 21:56:45.838995 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.838668 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-q2w2f" event={"ID":"442d5096-f83c-46f8-b4f8-79a0096899be","Type":"ContainerDied","Data":"e0fce1a23b4e41e4a6368feed1d35080b45ae808f497a6c64dfa9790bf72d9a6"} Jul 21 21:56:45.838995 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.838686 2557 scope.go:117] "RemoveContainer" containerID="7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d" Jul 21 21:56:45.847773 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.847754 2557 scope.go:117] "RemoveContainer" containerID="7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d" Jul 21 21:56:45.848038 ip-10-0-138-36 kubenswrapper[2557]: E0721 21:56:45.848018 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d\": container with ID starting with 7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d not found: ID does not exist" containerID="7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d" Jul 21 21:56:45.848078 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.848046 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d"} err="failed to get container status \"7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d\": rpc error: code = NotFound desc = could not find container \"7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d\": container with ID starting with 7695c592691dbc53bd55293fefe3d5b6541fdd8e45e08773dbdd290823ecf98d not found: ID does not exist" Jul 21 21:56:45.860519 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.860493 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-q2w2f"] Jul 21 21:56:45.864244 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:45.864223 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-q2w2f"] Jul 21 21:56:46.389705 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:46.389672 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="442d5096-f83c-46f8-b4f8-79a0096899be" path="/var/lib/kubelet/pods/442d5096-f83c-46f8-b4f8-79a0096899be/volumes" Jul 21 21:56:53.828528 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:56:53.828499 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-lvrlh" Jul 21 21:57:51.488744 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:57:51.488703 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:58:14.938833 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:14.938796 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:58:20.252542 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:20.252517 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 21:58:20.253001 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:20.252638 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 21:58:39.245682 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:39.245650 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:58:46.141832 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:46.141750 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:58:51.486835 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.486798 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h"] Jul 21 21:58:51.487245 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.487155 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="442d5096-f83c-46f8-b4f8-79a0096899be" containerName="authorino" Jul 21 21:58:51.487245 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.487167 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="442d5096-f83c-46f8-b4f8-79a0096899be" containerName="authorino" Jul 21 21:58:51.487245 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.487240 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="442d5096-f83c-46f8-b4f8-79a0096899be" containerName="authorino" Jul 21 21:58:51.490065 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.490045 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.492990 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.492966 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 21 21:58:51.493847 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.493830 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 21 21:58:51.493964 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.493872 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 21 21:58:51.493964 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.493875 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-kccl4\"" Jul 21 21:58:51.504345 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.504319 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h"] Jul 21 21:58:51.569376 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.569342 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.569376 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.569385 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.569601 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.569482 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.569601 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.569544 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6aae96-b948-4ae8-8510-2db9dcdaba23-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.569601 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.569563 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.569601 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.569586 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9ctc\" (UniqueName: \"kubernetes.io/projected/ac6aae96-b948-4ae8-8510-2db9dcdaba23-kube-api-access-x9ctc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670014 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.669972 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670250 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.670030 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6aae96-b948-4ae8-8510-2db9dcdaba23-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670250 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.670158 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670250 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.670192 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9ctc\" (UniqueName: \"kubernetes.io/projected/ac6aae96-b948-4ae8-8510-2db9dcdaba23-kube-api-access-x9ctc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670250 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.670212 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670250 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.670238 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670525 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.670483 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670569 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.670535 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.670706 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.670685 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.672491 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.672465 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ac6aae96-b948-4ae8-8510-2db9dcdaba23-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.672771 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.672750 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ac6aae96-b948-4ae8-8510-2db9dcdaba23-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.680085 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.680061 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9ctc\" (UniqueName: \"kubernetes.io/projected/ac6aae96-b948-4ae8-8510-2db9dcdaba23-kube-api-access-x9ctc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h\" (UID: \"ac6aae96-b948-4ae8-8510-2db9dcdaba23\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.801360 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.801269 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:58:51.946132 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.946082 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h"] Jul 21 21:58:51.947842 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:58:51.947818 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac6aae96_b948_4ae8_8510_2db9dcdaba23.slice/crio-69d8030d5302670ce005bd78293a84efee28bb22c399ff00bd9849e212545273 WatchSource:0}: Error finding container 69d8030d5302670ce005bd78293a84efee28bb22c399ff00bd9849e212545273: Status 404 returned error can't find the container with id 69d8030d5302670ce005bd78293a84efee28bb22c399ff00bd9849e212545273 Jul 21 21:58:51.949230 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:51.949214 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 21:58:52.477409 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:52.477373 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" event={"ID":"ac6aae96-b948-4ae8-8510-2db9dcdaba23","Type":"ContainerStarted","Data":"69d8030d5302670ce005bd78293a84efee28bb22c399ff00bd9849e212545273"} Jul 21 21:58:53.449060 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:53.449027 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:58:53.484842 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:53.484808 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" event={"ID":"ac6aae96-b948-4ae8-8510-2db9dcdaba23","Type":"ContainerStarted","Data":"8dc28096d590c2a6616621ade97a958638b14f7778ff4384808ac1a3e16abc5f"} Jul 21 21:58:53.507107 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:58:53.507034 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" podStartSLOduration=1.511741796 podStartE2EDuration="2.507017888s" totalImagesPullingTime="995.276092ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:58:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:58:53.504615916 +0000 UTC m=+633.610580061" watchObservedRunningTime="2026-07-21 21:58:53.507017888 +0000 UTC m=+633.612982087" Jul 21 21:59:01.801541 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:01.801505 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:59:01.801977 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:01.801579 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:59:01.817122 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:01.817075 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:59:02.541611 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:02.541582 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h" Jul 21 21:59:03.035172 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:03.035139 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:59:09.677515 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.677476 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz"] Jul 21 21:59:09.680829 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.680810 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.683662 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.683643 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 21 21:59:09.690874 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.690841 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz"] Jul 21 21:59:09.708250 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.708209 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.708428 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.708262 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.708428 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.708288 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f80ef63-43c1-432f-8942-618092eb6709-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.708428 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.708336 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxwsh\" (UniqueName: \"kubernetes.io/projected/5f80ef63-43c1-432f-8942-618092eb6709-kube-api-access-gxwsh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.708531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.708433 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.708531 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.708489 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.809192 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809157 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.809403 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809215 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.809403 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809238 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.809403 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809272 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.809403 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809294 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f80ef63-43c1-432f-8942-618092eb6709-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.809403 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809312 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxwsh\" (UniqueName: \"kubernetes.io/projected/5f80ef63-43c1-432f-8942-618092eb6709-kube-api-access-gxwsh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.810001 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809631 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.810001 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809671 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.810001 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.809689 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.811700 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.811675 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5f80ef63-43c1-432f-8942-618092eb6709-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.812179 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.812155 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5f80ef63-43c1-432f-8942-618092eb6709-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.818568 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.818539 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxwsh\" (UniqueName: \"kubernetes.io/projected/5f80ef63-43c1-432f-8942-618092eb6709-kube-api-access-gxwsh\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-x4szz\" (UID: \"5f80ef63-43c1-432f-8942-618092eb6709\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:09.990654 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:09.990616 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:10.136693 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:10.136656 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz"] Jul 21 21:59:10.139183 ip-10-0-138-36 kubenswrapper[2557]: W0721 21:59:10.139074 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f80ef63_43c1_432f_8942_618092eb6709.slice/crio-281960c1c1ac4e219d042066fa507f4f692f8814acb7f120d896b05612e513d4 WatchSource:0}: Error finding container 281960c1c1ac4e219d042066fa507f4f692f8814acb7f120d896b05612e513d4: Status 404 returned error can't find the container with id 281960c1c1ac4e219d042066fa507f4f692f8814acb7f120d896b05612e513d4 Jul 21 21:59:10.571400 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:10.571363 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" event={"ID":"5f80ef63-43c1-432f-8942-618092eb6709","Type":"ContainerStarted","Data":"3dedcf33603d3dd2584957af211cb46920e3cab6de34359842983c6f628dfc77"} Jul 21 21:59:10.571400 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:10.571412 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" event={"ID":"5f80ef63-43c1-432f-8942-618092eb6709","Type":"ContainerStarted","Data":"281960c1c1ac4e219d042066fa507f4f692f8814acb7f120d896b05612e513d4"} Jul 21 21:59:12.732454 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:12.732396 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" podStartSLOduration=3.507007915 podStartE2EDuration="3.732377661s" totalImagesPullingTime="225.369746ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 21:59:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 21:59:10.60813538 +0000 UTC m=+650.714099521" watchObservedRunningTime="2026-07-21 21:59:12.732377661 +0000 UTC m=+652.838341808" Jul 21 21:59:12.734517 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:12.734486 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 21:59:19.991136 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:19.991070 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:19.991580 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:19.991164 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:20.004865 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:20.004828 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 21:59:20.632520 ip-10-0-138-36 kubenswrapper[2557]: I0721 21:59:20.632488 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-x4szz" Jul 21 22:00:45.407332 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.407250 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6fd888dcb8-zdcgs"] Jul 21 22:00:45.410468 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.410449 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" Jul 21 22:00:45.413992 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.413961 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-t5swz\"" Jul 21 22:00:45.413992 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.413982 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Jul 21 22:00:45.420288 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.420263 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fd888dcb8-zdcgs"] Jul 21 22:00:45.454023 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.453989 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/defe2a6f-3e8b-4636-9549-1c4e1376c572-tls-cert\") pod \"authorino-6fd888dcb8-zdcgs\" (UID: \"defe2a6f-3e8b-4636-9549-1c4e1376c572\") " pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" Jul 21 22:00:45.454209 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.454054 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vf26z\" (UniqueName: \"kubernetes.io/projected/defe2a6f-3e8b-4636-9549-1c4e1376c572-kube-api-access-vf26z\") pod \"authorino-6fd888dcb8-zdcgs\" (UID: \"defe2a6f-3e8b-4636-9549-1c4e1376c572\") " pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" Jul 21 22:00:45.555017 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.554983 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/defe2a6f-3e8b-4636-9549-1c4e1376c572-tls-cert\") pod \"authorino-6fd888dcb8-zdcgs\" (UID: \"defe2a6f-3e8b-4636-9549-1c4e1376c572\") " pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" Jul 21 22:00:45.555194 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.555047 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vf26z\" (UniqueName: \"kubernetes.io/projected/defe2a6f-3e8b-4636-9549-1c4e1376c572-kube-api-access-vf26z\") pod \"authorino-6fd888dcb8-zdcgs\" (UID: \"defe2a6f-3e8b-4636-9549-1c4e1376c572\") " pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" Jul 21 22:00:45.557555 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.557528 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/defe2a6f-3e8b-4636-9549-1c4e1376c572-tls-cert\") pod \"authorino-6fd888dcb8-zdcgs\" (UID: \"defe2a6f-3e8b-4636-9549-1c4e1376c572\") " pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" Jul 21 22:00:45.564520 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.564495 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vf26z\" (UniqueName: \"kubernetes.io/projected/defe2a6f-3e8b-4636-9549-1c4e1376c572-kube-api-access-vf26z\") pod \"authorino-6fd888dcb8-zdcgs\" (UID: \"defe2a6f-3e8b-4636-9549-1c4e1376c572\") " pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" Jul 21 22:00:45.719995 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.719900 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" Jul 21 22:00:45.860385 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:45.860360 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6fd888dcb8-zdcgs"] Jul 21 22:00:45.862201 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:00:45.862172 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddefe2a6f_3e8b_4636_9549_1c4e1376c572.slice/crio-00201a322cbe38a7792d1556a68147794a00556936923d8e661ba04d1a3453f7 WatchSource:0}: Error finding container 00201a322cbe38a7792d1556a68147794a00556936923d8e661ba04d1a3453f7: Status 404 returned error can't find the container with id 00201a322cbe38a7792d1556a68147794a00556936923d8e661ba04d1a3453f7 Jul 21 22:00:46.060168 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:46.060129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" event={"ID":"defe2a6f-3e8b-4636-9549-1c4e1376c572","Type":"ContainerStarted","Data":"00201a322cbe38a7792d1556a68147794a00556936923d8e661ba04d1a3453f7"} Jul 21 22:00:47.067728 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:47.067691 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" event={"ID":"defe2a6f-3e8b-4636-9549-1c4e1376c572","Type":"ContainerStarted","Data":"c3b6512fee46adee40c0c9c8c5e2c6c50381c324f5a7599008a33be217dc4c87"} Jul 21 22:00:47.090931 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:00:47.090871 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6fd888dcb8-zdcgs" podStartSLOduration=1.602536706 podStartE2EDuration="2.09084704s" totalImagesPullingTime="488.310334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:00:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:00:47.090307255 +0000 UTC m=+747.196271403" watchObservedRunningTime="2026-07-21 22:00:47.09084704 +0000 UTC m=+747.196811187" Jul 21 22:01:18.294375 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:01:18.294340 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:01:28.765296 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:01:28.765264 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:01:38.419172 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:01:38.419131 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:03:20.293536 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:03:20.293447 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:03:20.294033 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:03:20.293954 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:03:22.871936 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:03:22.871904 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:03:31.452987 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:03:31.452949 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:04:09.439346 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:04:09.439308 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:04:18.642409 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:04:18.642368 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:04:27.341325 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:04:27.341273 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:04:33.442706 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:04:33.442662 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:04:44.140008 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:04:44.139904 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:04:53.552103 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:04:53.552053 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:05:05.236383 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:05:05.236339 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:05:15.134627 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:05:15.134592 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:05:40.649903 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:05:40.649866 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:05:50.238465 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:05:50.238433 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:06:15.837321 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:06:15.837228 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:06:26.044553 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:06:26.044517 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:06:41.555993 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:06:41.555956 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:06:50.459924 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:06:50.459884 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:07:01.639280 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:07:01.639239 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:07:11.075257 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:07:11.075225 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:07:20.350872 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:07:20.350824 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:08:20.333632 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:08:20.333599 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:08:20.334832 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:08:20.334809 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:09:10.173053 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:09:10.173017 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:09:21.849235 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:09:21.849151 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:09:29.848551 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:09:29.848512 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:09:40.347595 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:09:40.347561 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:09:51.055530 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:09:51.055489 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:10:02.765057 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:10:02.765023 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:10:11.747547 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:10:11.747509 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:10:23.652671 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:10:23.652637 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:10:33.245254 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:10:33.245205 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:10:40.239276 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:10:40.239244 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:10:49.251762 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:10:49.251732 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:10:58.734881 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:10:58.734850 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:11:16.740402 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:16.740367 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:11:25.933341 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:25.933309 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:11:36.680788 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.680751 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh"] Jul 21 22:11:36.684251 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.684234 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.686779 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.686762 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c461abb7-kserve-self-signed-certs\"" Jul 21 22:11:36.702972 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.702950 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh"] Jul 21 22:11:36.795490 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.795453 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-model-cache\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.795693 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.795523 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-home\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.795693 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.795552 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-dshm\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.795693 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.795583 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nxdr\" (UniqueName: \"kubernetes.io/projected/9d807221-b447-4372-9d13-8b130971cdef-kube-api-access-9nxdr\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.795693 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.795666 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-tmp-dir\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.795860 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.795702 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d807221-b447-4372-9d13-8b130971cdef-tls-certs\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.896800 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.896766 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-home\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.896800 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.896807 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-dshm\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.897051 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.896831 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nxdr\" (UniqueName: \"kubernetes.io/projected/9d807221-b447-4372-9d13-8b130971cdef-kube-api-access-9nxdr\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.897051 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.896863 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-tmp-dir\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.897051 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.896881 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d807221-b447-4372-9d13-8b130971cdef-tls-certs\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.897051 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.896922 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-model-cache\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.897360 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.897332 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-home\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.897424 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.897368 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-tmp-dir\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.897424 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.897413 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-model-cache\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.899248 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.899225 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-dshm\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.899662 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.899637 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d807221-b447-4372-9d13-8b130971cdef-tls-certs\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.904964 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.904944 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nxdr\" (UniqueName: \"kubernetes.io/projected/9d807221-b447-4372-9d13-8b130971cdef-kube-api-access-9nxdr\") pod \"e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:36.993237 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:36.993207 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:37.131451 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:37.131426 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh"] Jul 21 22:11:37.133261 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:11:37.133237 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9d807221_b447_4372_9d13_8b130971cdef.slice/crio-f44b9f151f133503fd42f137358c35cb8f0482df83b67b0ce700ffff4cb34653 WatchSource:0}: Error finding container f44b9f151f133503fd42f137358c35cb8f0482df83b67b0ce700ffff4cb34653: Status 404 returned error can't find the container with id f44b9f151f133503fd42f137358c35cb8f0482df83b67b0ce700ffff4cb34653 Jul 21 22:11:37.393576 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:37.393482 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" event={"ID":"9d807221-b447-4372-9d13-8b130971cdef","Type":"ContainerStarted","Data":"4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5"} Jul 21 22:11:37.393576 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:37.393523 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" event={"ID":"9d807221-b447-4372-9d13-8b130971cdef","Type":"ContainerStarted","Data":"f44b9f151f133503fd42f137358c35cb8f0482df83b67b0ce700ffff4cb34653"} Jul 21 22:11:37.411458 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:37.411414 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" podStartSLOduration=1.411397996 podStartE2EDuration="1.411397996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:11:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:11:37.410747617 +0000 UTC m=+1397.516711764" watchObservedRunningTime="2026-07-21 22:11:37.411397996 +0000 UTC m=+1397.517362142" Jul 21 22:11:38.931218 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:38.931183 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:11:46.993559 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:46.993525 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:46.994048 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:46.993591 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:47.006398 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:47.006371 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:47.455678 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:47.455592 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:11:53.739239 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:11:53.739204 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:12:12.008286 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:12.008203 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh"] Jul 21 22:12:16.031296 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:16.031264 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:12:27.029070 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.028999 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" podUID="9d807221-b447-4372-9d13-8b130971cdef" containerName="main" containerID="cri-o://4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5" gracePeriod=45 Jul 21 22:12:27.287429 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.287368 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:12:27.426987 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.426935 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-tmp-dir\") pod \"9d807221-b447-4372-9d13-8b130971cdef\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " Jul 21 22:12:27.426987 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.426996 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-home\") pod \"9d807221-b447-4372-9d13-8b130971cdef\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " Jul 21 22:12:27.427249 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427016 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nxdr\" (UniqueName: \"kubernetes.io/projected/9d807221-b447-4372-9d13-8b130971cdef-kube-api-access-9nxdr\") pod \"9d807221-b447-4372-9d13-8b130971cdef\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " Jul 21 22:12:27.427249 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427039 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-dshm\") pod \"9d807221-b447-4372-9d13-8b130971cdef\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " Jul 21 22:12:27.427249 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427148 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d807221-b447-4372-9d13-8b130971cdef-tls-certs\") pod \"9d807221-b447-4372-9d13-8b130971cdef\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " Jul 21 22:12:27.427249 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427206 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-model-cache\") pod \"9d807221-b447-4372-9d13-8b130971cdef\" (UID: \"9d807221-b447-4372-9d13-8b130971cdef\") " Jul 21 22:12:27.427461 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427314 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9d807221-b447-4372-9d13-8b130971cdef" (UID: "9d807221-b447-4372-9d13-8b130971cdef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:12:27.427461 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427324 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-home" (OuterVolumeSpecName: "home") pod "9d807221-b447-4372-9d13-8b130971cdef" (UID: "9d807221-b447-4372-9d13-8b130971cdef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:12:27.427564 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427470 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-tmp-dir\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:12:27.427564 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427483 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-home\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:12:27.427564 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.427520 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-model-cache" (OuterVolumeSpecName: "model-cache") pod "9d807221-b447-4372-9d13-8b130971cdef" (UID: "9d807221-b447-4372-9d13-8b130971cdef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:12:27.429550 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.429521 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d807221-b447-4372-9d13-8b130971cdef-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9d807221-b447-4372-9d13-8b130971cdef" (UID: "9d807221-b447-4372-9d13-8b130971cdef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:12:27.429652 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.429551 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-dshm" (OuterVolumeSpecName: "dshm") pod "9d807221-b447-4372-9d13-8b130971cdef" (UID: "9d807221-b447-4372-9d13-8b130971cdef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:12:27.429652 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.429568 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d807221-b447-4372-9d13-8b130971cdef-kube-api-access-9nxdr" (OuterVolumeSpecName: "kube-api-access-9nxdr") pod "9d807221-b447-4372-9d13-8b130971cdef" (UID: "9d807221-b447-4372-9d13-8b130971cdef"). InnerVolumeSpecName "kube-api-access-9nxdr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:12:27.528186 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.528146 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9d807221-b447-4372-9d13-8b130971cdef-tls-certs\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:12:27.528186 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.528181 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-model-cache\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:12:27.528186 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.528190 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nxdr\" (UniqueName: \"kubernetes.io/projected/9d807221-b447-4372-9d13-8b130971cdef-kube-api-access-9nxdr\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:12:27.528427 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.528198 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9d807221-b447-4372-9d13-8b130971cdef-dshm\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:12:27.655623 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.655536 2557 generic.go:356] "Generic (PLEG): container finished" podID="9d807221-b447-4372-9d13-8b130971cdef" containerID="4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5" exitCode=0 Jul 21 22:12:27.655623 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.655584 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" Jul 21 22:12:27.655849 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.655627 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" event={"ID":"9d807221-b447-4372-9d13-8b130971cdef","Type":"ContainerDied","Data":"4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5"} Jul 21 22:12:27.655849 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.655679 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh" event={"ID":"9d807221-b447-4372-9d13-8b130971cdef","Type":"ContainerDied","Data":"f44b9f151f133503fd42f137358c35cb8f0482df83b67b0ce700ffff4cb34653"} Jul 21 22:12:27.655849 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.655703 2557 scope.go:117] "RemoveContainer" containerID="4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5" Jul 21 22:12:27.665354 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.665339 2557 scope.go:117] "RemoveContainer" containerID="4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5" Jul 21 22:12:27.665597 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:12:27.665580 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5\": container with ID starting with 4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5 not found: ID does not exist" containerID="4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5" Jul 21 22:12:27.665645 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.665605 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5"} err="failed to get container status \"4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5\": rpc error: code = NotFound desc = could not find container \"4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5\": container with ID starting with 4ed02392ebc9636b6c2fe9e69ae699da8319870c43b88e395d4d9472a62983b5 not found: ID does not exist" Jul 21 22:12:27.676669 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.676646 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh"] Jul 21 22:12:27.680447 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:27.680425 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c461abb7-kserve-c5fc9794c-pq6zh"] Jul 21 22:12:28.390081 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:28.390047 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d807221-b447-4372-9d13-8b130971cdef" path="/var/lib/kubelet/pods/9d807221-b447-4372-9d13-8b130971cdef/volumes" Jul 21 22:12:29.926345 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:29.926311 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:12:37.329312 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:37.329275 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:12:46.428911 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:46.428873 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:12:54.335127 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:12:54.335075 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:13:03.835770 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:13:03.835737 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:13:13.626351 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:13:13.626317 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:13:20.370707 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:13:20.370681 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:13:20.373650 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:13:20.373633 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:13:22.935069 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:13:22.935035 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:13:32.431753 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:13:32.431716 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:13:42.130209 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:13:42.130124 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:13:56.036630 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:13:56.036589 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:14:05.338501 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:14:05.338468 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:14:14.633724 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:14:14.633689 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:14:24.567949 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:14:24.567917 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:14:44.252678 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:14:44.252628 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:14:54.442082 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:14:54.442041 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:15:10.188620 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:15:10.188574 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:15:19.630375 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:15:19.630299 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:15:33.432300 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:15:33.432266 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:15:42.932997 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:15:42.932962 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:15:51.726891 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:15:51.726847 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:15:59.531136 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:15:59.531077 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:16:34.934567 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:16:34.934537 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:16:45.560437 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:16:45.560348 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:17:18.335824 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:17:18.335787 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:17:29.328668 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:17:29.328625 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:18:02.415315 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:18:02.415280 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:18:11.641695 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:18:11.641656 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:18:20.401723 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:18:20.401699 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:18:20.407897 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:18:20.407873 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:18:46.532240 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:18:46.532204 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:20:14.642937 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:20:14.642892 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:20:27.746998 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:20:27.746959 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:20:42.628087 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:20:42.628051 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:20:56.038936 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:20:56.038900 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:21:30.644500 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:30.644409 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:21:43.086862 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.086831 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt"] Jul 21 22:21:43.087254 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.087204 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d807221-b447-4372-9d13-8b130971cdef" containerName="main" Jul 21 22:21:43.087254 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.087217 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d807221-b447-4372-9d13-8b130971cdef" containerName="main" Jul 21 22:21:43.087327 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.087274 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d807221-b447-4372-9d13-8b130971cdef" containerName="main" Jul 21 22:21:43.090312 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.090294 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.094404 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.094384 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6699167f\"/\"default-dockercfg-fv8gq\"" Jul 21 22:21:43.094525 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.094411 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6699167f\"/\"openshift-service-ca.crt\"" Jul 21 22:21:43.094525 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.094411 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-6699167f\"/\"e2e-lifecycle-model-6699167f-kserve-self-signed-certs\"" Jul 21 22:21:43.094525 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.094385 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-6699167f\"/\"kube-root-ca.crt\"" Jul 21 22:21:43.101598 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.101575 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt"] Jul 21 22:21:43.196123 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.196082 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-dshm\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.196227 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.196139 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f9cb17-47ac-43ff-b215-5cae26e83d90-tls-certs\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.196227 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.196178 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-tmp-dir\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.196227 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.196196 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-home\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.196330 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.196230 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-model-cache\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.196330 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.196268 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm7vn\" (UniqueName: \"kubernetes.io/projected/59f9cb17-47ac-43ff-b215-5cae26e83d90-kube-api-access-wm7vn\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.296724 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.296700 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm7vn\" (UniqueName: \"kubernetes.io/projected/59f9cb17-47ac-43ff-b215-5cae26e83d90-kube-api-access-wm7vn\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.296827 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.296749 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-dshm\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.296827 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.296773 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f9cb17-47ac-43ff-b215-5cae26e83d90-tls-certs\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.296827 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.296792 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-tmp-dir\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.296827 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.296810 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-home\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.296993 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.296833 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-model-cache\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.297242 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.297218 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-home\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.297317 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.297277 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-model-cache\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.297358 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.297314 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-tmp-dir\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.298984 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.298963 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-dshm\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.299256 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.299238 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f9cb17-47ac-43ff-b215-5cae26e83d90-tls-certs\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.306555 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.306535 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm7vn\" (UniqueName: \"kubernetes.io/projected/59f9cb17-47ac-43ff-b215-5cae26e83d90-kube-api-access-wm7vn\") pod \"e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.398798 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.398730 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:43.534648 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:43.534615 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt"] Jul 21 22:21:43.537559 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:21:43.537529 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59f9cb17_47ac_43ff_b215_5cae26e83d90.slice/crio-6257cdd6da87cfa56cd8cbc98b913604d264cc70eb638e6aa4b061067796c59d WatchSource:0}: Error finding container 6257cdd6da87cfa56cd8cbc98b913604d264cc70eb638e6aa4b061067796c59d: Status 404 returned error can't find the container with id 6257cdd6da87cfa56cd8cbc98b913604d264cc70eb638e6aa4b061067796c59d Jul 21 22:21:44.456396 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:44.456355 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" event={"ID":"59f9cb17-47ac-43ff-b215-5cae26e83d90","Type":"ContainerStarted","Data":"d77ace9d4eba3a03d41e544e60960f5e9ed62b026cc7cf0587c94a3ee17fe3d8"} Jul 21 22:21:44.456777 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:44.456407 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" event={"ID":"59f9cb17-47ac-43ff-b215-5cae26e83d90","Type":"ContainerStarted","Data":"6257cdd6da87cfa56cd8cbc98b913604d264cc70eb638e6aa4b061067796c59d"} Jul 21 22:21:44.475371 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:44.475327 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" podStartSLOduration=1.475312621 podStartE2EDuration="1.475312621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:21:44.474811148 +0000 UTC m=+2004.580775294" watchObservedRunningTime="2026-07-21 22:21:44.475312621 +0000 UTC m=+2004.581276771" Jul 21 22:21:47.439155 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:47.439120 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:21:53.399554 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:53.399524 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:53.399922 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:53.399565 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:53.411447 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:53.411418 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:21:53.512249 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:21:53.512221 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:22:07.639714 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:07.639677 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:22:26.434895 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:26.434861 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:22:35.134691 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:35.134653 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:22:38.609357 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:38.609318 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt"] Jul 21 22:22:53.629938 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.629846 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" podUID="59f9cb17-47ac-43ff-b215-5cae26e83d90" containerName="main" containerID="cri-o://d77ace9d4eba3a03d41e544e60960f5e9ed62b026cc7cf0587c94a3ee17fe3d8" gracePeriod=45 Jul 21 22:22:53.814990 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.814962 2557 generic.go:356] "Generic (PLEG): container finished" podID="59f9cb17-47ac-43ff-b215-5cae26e83d90" containerID="d77ace9d4eba3a03d41e544e60960f5e9ed62b026cc7cf0587c94a3ee17fe3d8" exitCode=0 Jul 21 22:22:53.815143 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.815037 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" event={"ID":"59f9cb17-47ac-43ff-b215-5cae26e83d90","Type":"ContainerDied","Data":"d77ace9d4eba3a03d41e544e60960f5e9ed62b026cc7cf0587c94a3ee17fe3d8"} Jul 21 22:22:53.880203 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.880142 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:22:53.960215 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960184 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm7vn\" (UniqueName: \"kubernetes.io/projected/59f9cb17-47ac-43ff-b215-5cae26e83d90-kube-api-access-wm7vn\") pod \"59f9cb17-47ac-43ff-b215-5cae26e83d90\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " Jul 21 22:22:53.960374 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960247 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-tmp-dir\") pod \"59f9cb17-47ac-43ff-b215-5cae26e83d90\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " Jul 21 22:22:53.960374 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960268 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-dshm\") pod \"59f9cb17-47ac-43ff-b215-5cae26e83d90\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " Jul 21 22:22:53.960374 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960282 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-model-cache\") pod \"59f9cb17-47ac-43ff-b215-5cae26e83d90\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " Jul 21 22:22:53.960374 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960327 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f9cb17-47ac-43ff-b215-5cae26e83d90-tls-certs\") pod \"59f9cb17-47ac-43ff-b215-5cae26e83d90\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " Jul 21 22:22:53.960664 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960376 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-home\") pod \"59f9cb17-47ac-43ff-b215-5cae26e83d90\" (UID: \"59f9cb17-47ac-43ff-b215-5cae26e83d90\") " Jul 21 22:22:53.960664 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960546 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "59f9cb17-47ac-43ff-b215-5cae26e83d90" (UID: "59f9cb17-47ac-43ff-b215-5cae26e83d90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:22:53.960664 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960540 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-model-cache" (OuterVolumeSpecName: "model-cache") pod "59f9cb17-47ac-43ff-b215-5cae26e83d90" (UID: "59f9cb17-47ac-43ff-b215-5cae26e83d90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:22:53.960664 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960658 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-tmp-dir\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:22:53.960839 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960678 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-model-cache\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:22:53.960839 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.960704 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-home" (OuterVolumeSpecName: "home") pod "59f9cb17-47ac-43ff-b215-5cae26e83d90" (UID: "59f9cb17-47ac-43ff-b215-5cae26e83d90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:22:53.962504 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.962472 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59f9cb17-47ac-43ff-b215-5cae26e83d90-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "59f9cb17-47ac-43ff-b215-5cae26e83d90" (UID: "59f9cb17-47ac-43ff-b215-5cae26e83d90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:22:53.962504 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.962500 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-dshm" (OuterVolumeSpecName: "dshm") pod "59f9cb17-47ac-43ff-b215-5cae26e83d90" (UID: "59f9cb17-47ac-43ff-b215-5cae26e83d90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:22:53.962646 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:53.962504 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f9cb17-47ac-43ff-b215-5cae26e83d90-kube-api-access-wm7vn" (OuterVolumeSpecName: "kube-api-access-wm7vn") pod "59f9cb17-47ac-43ff-b215-5cae26e83d90" (UID: "59f9cb17-47ac-43ff-b215-5cae26e83d90"). InnerVolumeSpecName "kube-api-access-wm7vn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:22:54.061343 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.061319 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-home\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:22:54.061343 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.061341 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm7vn\" (UniqueName: \"kubernetes.io/projected/59f9cb17-47ac-43ff-b215-5cae26e83d90-kube-api-access-wm7vn\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:22:54.061484 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.061351 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59f9cb17-47ac-43ff-b215-5cae26e83d90-dshm\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:22:54.061484 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.061360 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59f9cb17-47ac-43ff-b215-5cae26e83d90-tls-certs\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:22:54.822298 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.822267 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" Jul 21 22:22:54.822709 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.822306 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt" event={"ID":"59f9cb17-47ac-43ff-b215-5cae26e83d90","Type":"ContainerDied","Data":"6257cdd6da87cfa56cd8cbc98b913604d264cc70eb638e6aa4b061067796c59d"} Jul 21 22:22:54.822709 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.822357 2557 scope.go:117] "RemoveContainer" containerID="d77ace9d4eba3a03d41e544e60960f5e9ed62b026cc7cf0587c94a3ee17fe3d8" Jul 21 22:22:54.848259 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.848236 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt"] Jul 21 22:22:54.854626 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:54.854603 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-6699167f/e2e-lifecycle-model-6699167f-kserve-86dd8dc995-k2qrt"] Jul 21 22:22:56.389143 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:22:56.389108 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59f9cb17-47ac-43ff-b215-5cae26e83d90" path="/var/lib/kubelet/pods/59f9cb17-47ac-43ff-b215-5cae26e83d90/volumes" Jul 21 22:23:20.436653 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:23:20.436629 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:23:20.444444 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:23:20.444424 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:23:26.233972 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:23:26.233934 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:23:35.936849 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:23:35.936814 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:24:26.811266 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:24:26.811187 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:24:42.725923 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:24:42.725881 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:25:00.564484 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.564452 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl"] Jul 21 22:25:00.565238 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.564772 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59f9cb17-47ac-43ff-b215-5cae26e83d90" containerName="main" Jul 21 22:25:00.565238 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.564783 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="59f9cb17-47ac-43ff-b215-5cae26e83d90" containerName="main" Jul 21 22:25:00.565238 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.564849 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="59f9cb17-47ac-43ff-b215-5cae26e83d90" containerName="main" Jul 21 22:25:00.567722 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.567705 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.573020 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.572995 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3f5ae6\"/\"default-dockercfg-l5fgd\"" Jul 21 22:25:00.573168 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.573004 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3f5ae6\"/\"openshift-service-ca.crt\"" Jul 21 22:25:00.573168 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.573003 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-3f5ae6\"/\"kube-root-ca.crt\"" Jul 21 22:25:00.573168 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.573105 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-3f5ae6\"/\"test-model-3f5ae6-kserve-self-signed-certs\"" Jul 21 22:25:00.586991 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.586936 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl"] Jul 21 22:25:00.603798 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.603764 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fl45\" (UniqueName: \"kubernetes.io/projected/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-kube-api-access-4fl45\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.603798 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.603803 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-home\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.604036 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.603890 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tls-certs\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.604036 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.603927 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-model-cache\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.604036 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.603976 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-dshm\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.604036 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.604000 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tmp-dir\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.704455 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.704416 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tls-certs\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.704455 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.704455 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-model-cache\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.704697 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.704486 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-dshm\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.704697 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.704505 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tmp-dir\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.704697 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.704543 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fl45\" (UniqueName: \"kubernetes.io/projected/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-kube-api-access-4fl45\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.704697 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.704577 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-home\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.704902 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.704876 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-model-cache\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.704972 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.704952 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-home\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.705029 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.705017 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tmp-dir\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.707021 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.707001 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-dshm\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.707219 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.707201 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tls-certs\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.714450 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.714424 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fl45\" (UniqueName: \"kubernetes.io/projected/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-kube-api-access-4fl45\") pod \"test-model-3f5ae6-kserve-7594b49557-rccrl\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:00.877353 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:00.877245 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:01.014756 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:01.014727 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl"] Jul 21 22:25:01.016652 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:25:01.016621 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod306cfcdc_ba35_4708_ba41_b9b45c8c9c64.slice/crio-ec1961814a7b525c2ba8b5157f8cb87ba8475414fd2fff1a9d5c63fee72690a9 WatchSource:0}: Error finding container ec1961814a7b525c2ba8b5157f8cb87ba8475414fd2fff1a9d5c63fee72690a9: Status 404 returned error can't find the container with id ec1961814a7b525c2ba8b5157f8cb87ba8475414fd2fff1a9d5c63fee72690a9 Jul 21 22:25:01.445645 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:01.445607 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" event={"ID":"306cfcdc-ba35-4708-ba41-b9b45c8c9c64","Type":"ContainerStarted","Data":"d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4"} Jul 21 22:25:01.445645 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:01.445653 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" event={"ID":"306cfcdc-ba35-4708-ba41-b9b45c8c9c64","Type":"ContainerStarted","Data":"ec1961814a7b525c2ba8b5157f8cb87ba8475414fd2fff1a9d5c63fee72690a9"} Jul 21 22:25:01.467475 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:01.467416 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" podStartSLOduration=1.46740092 podStartE2EDuration="1.46740092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:25:01.465917339 +0000 UTC m=+2201.571881484" watchObservedRunningTime="2026-07-21 22:25:01.46740092 +0000 UTC m=+2201.573365066" Jul 21 22:25:04.326129 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:04.326078 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:25:10.878056 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:10.878019 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:10.878668 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:10.878075 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:10.890492 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:10.890465 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:11.508806 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:11.508776 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:25:15.426382 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:15.426342 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:25:41.032629 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:25:41.032544 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:26:20.115749 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:20.115713 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl"] Jul 21 22:26:24.436999 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:24.436963 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:26:35.136461 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.136390 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" podUID="306cfcdc-ba35-4708-ba41-b9b45c8c9c64" containerName="main" containerID="cri-o://d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4" gracePeriod=45 Jul 21 22:26:35.392918 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.392864 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:26:35.477539 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.477503 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-model-cache\") pod \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " Jul 21 22:26:35.477717 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.477588 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-home\") pod \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " Jul 21 22:26:35.477717 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.477646 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tls-certs\") pod \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " Jul 21 22:26:35.477717 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.477668 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-dshm\") pod \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " Jul 21 22:26:35.477717 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.477709 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tmp-dir\") pod \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " Jul 21 22:26:35.477932 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.477764 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4fl45\" (UniqueName: \"kubernetes.io/projected/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-kube-api-access-4fl45\") pod \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\" (UID: \"306cfcdc-ba35-4708-ba41-b9b45c8c9c64\") " Jul 21 22:26:35.477932 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.477772 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-model-cache" (OuterVolumeSpecName: "model-cache") pod "306cfcdc-ba35-4708-ba41-b9b45c8c9c64" (UID: "306cfcdc-ba35-4708-ba41-b9b45c8c9c64"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:35.477932 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.477862 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-home" (OuterVolumeSpecName: "home") pod "306cfcdc-ba35-4708-ba41-b9b45c8c9c64" (UID: "306cfcdc-ba35-4708-ba41-b9b45c8c9c64"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:35.478119 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.478036 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "306cfcdc-ba35-4708-ba41-b9b45c8c9c64" (UID: "306cfcdc-ba35-4708-ba41-b9b45c8c9c64"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:35.478176 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.478131 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-model-cache\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:26:35.478176 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.478150 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-home\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:26:35.480075 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.480050 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-dshm" (OuterVolumeSpecName: "dshm") pod "306cfcdc-ba35-4708-ba41-b9b45c8c9c64" (UID: "306cfcdc-ba35-4708-ba41-b9b45c8c9c64"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:26:35.480075 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.480060 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-kube-api-access-4fl45" (OuterVolumeSpecName: "kube-api-access-4fl45") pod "306cfcdc-ba35-4708-ba41-b9b45c8c9c64" (UID: "306cfcdc-ba35-4708-ba41-b9b45c8c9c64"). InnerVolumeSpecName "kube-api-access-4fl45". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:26:35.480232 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.480056 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "306cfcdc-ba35-4708-ba41-b9b45c8c9c64" (UID: "306cfcdc-ba35-4708-ba41-b9b45c8c9c64"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:26:35.579449 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.579422 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tls-certs\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:26:35.579449 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.579447 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-dshm\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:26:35.579449 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.579457 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-tmp-dir\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:26:35.579661 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.579466 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4fl45\" (UniqueName: \"kubernetes.io/projected/306cfcdc-ba35-4708-ba41-b9b45c8c9c64-kube-api-access-4fl45\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:26:35.933442 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.933408 2557 generic.go:356] "Generic (PLEG): container finished" podID="306cfcdc-ba35-4708-ba41-b9b45c8c9c64" containerID="d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4" exitCode=0 Jul 21 22:26:35.933643 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.933452 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" Jul 21 22:26:35.933643 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.933498 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" event={"ID":"306cfcdc-ba35-4708-ba41-b9b45c8c9c64","Type":"ContainerDied","Data":"d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4"} Jul 21 22:26:35.933643 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.933549 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl" event={"ID":"306cfcdc-ba35-4708-ba41-b9b45c8c9c64","Type":"ContainerDied","Data":"ec1961814a7b525c2ba8b5157f8cb87ba8475414fd2fff1a9d5c63fee72690a9"} Jul 21 22:26:35.933643 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.933571 2557 scope.go:117] "RemoveContainer" containerID="d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4" Jul 21 22:26:35.943059 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.943040 2557 scope.go:117] "RemoveContainer" containerID="d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4" Jul 21 22:26:35.943362 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:26:35.943343 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4\": container with ID starting with d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4 not found: ID does not exist" containerID="d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4" Jul 21 22:26:35.943439 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.943369 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4"} err="failed to get container status \"d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4\": rpc error: code = NotFound desc = could not find container \"d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4\": container with ID starting with d5c04f231c87b583748b1eb2a01f816705ffba295eea1c0dd7c26b6e1f387df4 not found: ID does not exist" Jul 21 22:26:35.955975 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.955946 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl"] Jul 21 22:26:35.959707 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:35.959684 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-3f5ae6/test-model-3f5ae6-kserve-7594b49557-rccrl"] Jul 21 22:26:36.391693 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:36.391663 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="306cfcdc-ba35-4708-ba41-b9b45c8c9c64" path="/var/lib/kubelet/pods/306cfcdc-ba35-4708-ba41-b9b45c8c9c64/volumes" Jul 21 22:26:37.832982 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:37.832954 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:26:56.531133 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:26:56.531079 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:27:15.727049 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:27:15.726962 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:27:31.933659 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:27:31.933616 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:27:48.431193 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:27:48.431161 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:28:17.255454 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:17.255413 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:28:20.469043 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:20.469011 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:28:20.477101 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:20.477068 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:28:31.743021 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:31.742988 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:28:41.938422 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:41.938345 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:28:53.494867 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.494835 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l"] Jul 21 22:28:53.495418 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.495392 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="306cfcdc-ba35-4708-ba41-b9b45c8c9c64" containerName="main" Jul 21 22:28:53.495418 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.495411 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="306cfcdc-ba35-4708-ba41-b9b45c8c9c64" containerName="main" Jul 21 22:28:53.495515 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.495505 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="306cfcdc-ba35-4708-ba41-b9b45c8c9c64" containerName="main" Jul 21 22:28:53.498918 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.498902 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.502514 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.502492 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"default-dockercfg-d8nnn\"" Jul 21 22:28:53.502514 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.502504 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"kube-root-ca.crt\"" Jul 21 22:28:53.502712 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.502517 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"auth-test-model-131f31-kserve-self-signed-certs\"" Jul 21 22:28:53.502712 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.502494 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"openshift-service-ca.crt\"" Jul 21 22:28:53.511281 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.511257 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l"] Jul 21 22:28:53.620851 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.620809 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tls-certs\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.621050 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.620882 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-dshm\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.621050 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.620926 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-home\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.621050 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.620963 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-model-cache\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.621050 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.620981 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tmp-dir\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.621050 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.620999 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzzpd\" (UniqueName: \"kubernetes.io/projected/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-kube-api-access-kzzpd\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.721477 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.721432 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-dshm\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.721660 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.721500 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-home\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.721660 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.721539 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-model-cache\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.721660 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.721558 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tmp-dir\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.721660 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.721584 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzzpd\" (UniqueName: \"kubernetes.io/projected/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-kube-api-access-kzzpd\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.721860 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.721710 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tls-certs\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.721963 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.721945 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-home\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.722036 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.721973 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-model-cache\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.722111 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.722036 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tmp-dir\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.723959 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.723938 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-dshm\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.724414 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.724391 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tls-certs\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.732128 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.732080 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzzpd\" (UniqueName: \"kubernetes.io/projected/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-kube-api-access-kzzpd\") pod \"auth-test-model-131f31-kserve-8644bc57cf-7tj2l\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.808630 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.808535 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:28:53.946781 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:53.946749 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l"] Jul 21 22:28:53.949670 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:28:53.949645 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4cc116f5_c84f_482f_a9c4_ed62bacf6e9b.slice/crio-a01b6cc0ce620a90fea0e6cd2d218b6f7c8b80f731a524f2e7bc0475a56e8d23 WatchSource:0}: Error finding container a01b6cc0ce620a90fea0e6cd2d218b6f7c8b80f731a524f2e7bc0475a56e8d23: Status 404 returned error can't find the container with id a01b6cc0ce620a90fea0e6cd2d218b6f7c8b80f731a524f2e7bc0475a56e8d23 Jul 21 22:28:54.619612 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:54.619576 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" event={"ID":"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b","Type":"ContainerStarted","Data":"5a95e699dfbf770c08077c183e5d66846ef0755550a079454729335faaf3bdb6"} Jul 21 22:28:54.619612 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:54.619616 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" event={"ID":"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b","Type":"ContainerStarted","Data":"a01b6cc0ce620a90fea0e6cd2d218b6f7c8b80f731a524f2e7bc0475a56e8d23"} Jul 21 22:28:54.640224 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:54.640169 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" podStartSLOduration=1.640153727 podStartE2EDuration="1.640153727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:28:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:28:54.639405992 +0000 UTC m=+2434.745370137" watchObservedRunningTime="2026-07-21 22:28:54.640153727 +0000 UTC m=+2434.746117903" Jul 21 22:28:58.240275 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:28:58.236541 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:29:03.809410 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:03.809373 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:29:03.809877 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:03.809666 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:29:03.821903 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:03.821880 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:29:04.681560 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:04.681533 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:29:19.982840 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:19.982775 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf"] Jul 21 22:29:19.986665 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:19.986643 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:19.990367 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:19.990337 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"auth-test-model-b77603-kserve-self-signed-certs\"" Jul 21 22:29:19.990518 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:19.990406 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"default-dockercfg-p4lx2\"" Jul 21 22:29:19.990518 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:19.990406 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"kube-root-ca.crt\"" Jul 21 22:29:19.990631 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:19.990406 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"openshift-service-ca.crt\"" Jul 21 22:29:19.996839 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:19.996813 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf"] Jul 21 22:29:20.149372 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.149331 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-dshm\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.149547 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.149387 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tls-certs\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.149547 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.149455 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-model-cache\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.149547 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.149517 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-home\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.149547 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.149540 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tmp-dir\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.149686 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.149565 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zr7b5\" (UniqueName: \"kubernetes.io/projected/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-kube-api-access-zr7b5\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250282 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250213 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-home\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250282 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250243 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tmp-dir\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250282 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250269 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zr7b5\" (UniqueName: \"kubernetes.io/projected/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-kube-api-access-zr7b5\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250540 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250320 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-dshm\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250540 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250357 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tls-certs\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250540 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250392 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-model-cache\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250697 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250605 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tmp-dir\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250697 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250649 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-home\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.250697 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.250676 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-model-cache\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.252684 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.252664 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-dshm\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.252851 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.252838 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"auth-test-model-b77603-kserve-self-signed-certs\"" Jul 21 22:29:20.258935 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.258916 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"kube-root-ca.crt\"" Jul 21 22:29:20.262914 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.262897 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tls-certs\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.270166 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.270149 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"openshift-service-ca.crt\"" Jul 21 22:29:20.280620 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.280602 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zr7b5\" (UniqueName: \"kubernetes.io/projected/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-kube-api-access-zr7b5\") pod \"auth-test-model-b77603-kserve-c78c8f4f9-krgvf\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.301492 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.301466 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"default-dockercfg-p4lx2\"" Jul 21 22:29:20.309687 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.309665 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:20.448285 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.448259 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf"] Jul 21 22:29:20.450228 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:29:20.450201 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod43554f7e_f3db_4c46_9b36_25cb03b2f1d7.slice/crio-444251428ce49a14fdedec0d23768f6a6811b74fc279181b3440223b699053bc WatchSource:0}: Error finding container 444251428ce49a14fdedec0d23768f6a6811b74fc279181b3440223b699053bc: Status 404 returned error can't find the container with id 444251428ce49a14fdedec0d23768f6a6811b74fc279181b3440223b699053bc Jul 21 22:29:20.749039 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.748999 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" event={"ID":"43554f7e-f3db-4c46-9b36-25cb03b2f1d7","Type":"ContainerStarted","Data":"8855f500daded7de82c5a6b32ab78c8d3b327f25a9d7596639fd60f22b1fb5ab"} Jul 21 22:29:20.749245 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.749049 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" event={"ID":"43554f7e-f3db-4c46-9b36-25cb03b2f1d7","Type":"ContainerStarted","Data":"444251428ce49a14fdedec0d23768f6a6811b74fc279181b3440223b699053bc"} Jul 21 22:29:20.770493 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:20.770399 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" podStartSLOduration=1.7703836640000001 podStartE2EDuration="1.770383664s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:29:20.768264421 +0000 UTC m=+2460.874228568" watchObservedRunningTime="2026-07-21 22:29:20.770383664 +0000 UTC m=+2460.876347811" Jul 21 22:29:24.029650 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:24.029614 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:29:30.310707 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:30.310668 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:30.311202 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:30.310724 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:30.323274 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:30.323251 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:30.810843 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:30.810816 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:29:38.232719 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:38.232683 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:29:56.455243 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:29:56.455204 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:30:12.239856 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:30:12.239776 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:30:30.731964 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:30:30.731927 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:30:50.029679 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:30:50.029641 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:31:08.148255 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:08.148215 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:31:19.198816 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:19.198783 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l"] Jul 21 22:31:20.237191 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:20.237152 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf"] Jul 21 22:31:23.736290 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:23.736248 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:31:34.219762 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.219722 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" podUID="4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" containerName="main" containerID="cri-o://5a95e699dfbf770c08077c183e5d66846ef0755550a079454729335faaf3bdb6" gracePeriod=45 Jul 21 22:31:34.439989 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.439957 2557 generic.go:356] "Generic (PLEG): container finished" podID="4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" containerID="5a95e699dfbf770c08077c183e5d66846ef0755550a079454729335faaf3bdb6" exitCode=0 Jul 21 22:31:34.440190 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.440036 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" event={"ID":"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b","Type":"ContainerDied","Data":"5a95e699dfbf770c08077c183e5d66846ef0755550a079454729335faaf3bdb6"} Jul 21 22:31:34.464605 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.464582 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:31:34.587258 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587227 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-model-cache\") pod \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " Jul 21 22:31:34.587450 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587269 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tmp-dir\") pod \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " Jul 21 22:31:34.587450 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587290 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-dshm\") pod \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " Jul 21 22:31:34.587450 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587361 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tls-certs\") pod \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " Jul 21 22:31:34.587450 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587395 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kzzpd\" (UniqueName: \"kubernetes.io/projected/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-kube-api-access-kzzpd\") pod \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " Jul 21 22:31:34.587450 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587430 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-home\") pod \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\" (UID: \"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b\") " Jul 21 22:31:34.587788 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587510 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-model-cache" (OuterVolumeSpecName: "model-cache") pod "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" (UID: "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:31:34.587788 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587551 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" (UID: "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:31:34.587788 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587731 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-home" (OuterVolumeSpecName: "home") pod "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" (UID: "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:31:34.587959 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587790 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-model-cache\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:34.587959 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.587824 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tmp-dir\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:34.589748 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.589704 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-dshm" (OuterVolumeSpecName: "dshm") pod "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" (UID: "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:31:34.589848 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.589778 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-kube-api-access-kzzpd" (OuterVolumeSpecName: "kube-api-access-kzzpd") pod "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" (UID: "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b"). InnerVolumeSpecName "kube-api-access-kzzpd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:31:34.589848 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.589781 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" (UID: "4cc116f5-c84f-482f-a9c4-ed62bacf6e9b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:31:34.688517 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.688487 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-home\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:34.688517 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.688511 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-dshm\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:34.688517 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.688519 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-tls-certs\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:34.688743 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:34.688529 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzzpd\" (UniqueName: \"kubernetes.io/projected/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b-kube-api-access-kzzpd\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:35.257747 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.257699 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" podUID="43554f7e-f3db-4c46-9b36-25cb03b2f1d7" containerName="main" containerID="cri-o://8855f500daded7de82c5a6b32ab78c8d3b327f25a9d7596639fd60f22b1fb5ab" gracePeriod=45 Jul 21 22:31:35.449902 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.449869 2557 generic.go:356] "Generic (PLEG): container finished" podID="43554f7e-f3db-4c46-9b36-25cb03b2f1d7" containerID="8855f500daded7de82c5a6b32ab78c8d3b327f25a9d7596639fd60f22b1fb5ab" exitCode=0 Jul 21 22:31:35.450072 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.449942 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" event={"ID":"43554f7e-f3db-4c46-9b36-25cb03b2f1d7","Type":"ContainerDied","Data":"8855f500daded7de82c5a6b32ab78c8d3b327f25a9d7596639fd60f22b1fb5ab"} Jul 21 22:31:35.451513 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.451483 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" event={"ID":"4cc116f5-c84f-482f-a9c4-ed62bacf6e9b","Type":"ContainerDied","Data":"a01b6cc0ce620a90fea0e6cd2d218b6f7c8b80f731a524f2e7bc0475a56e8d23"} Jul 21 22:31:35.451624 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.451515 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l" Jul 21 22:31:35.451624 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.451526 2557 scope.go:117] "RemoveContainer" containerID="5a95e699dfbf770c08077c183e5d66846ef0755550a079454729335faaf3bdb6" Jul 21 22:31:35.476953 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.476927 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l"] Jul 21 22:31:35.479893 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.479870 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/auth-test-model-131f31-kserve-8644bc57cf-7tj2l"] Jul 21 22:31:35.519371 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.519310 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:31:35.595379 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595344 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-model-cache\") pod \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " Jul 21 22:31:35.595379 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595382 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-home\") pod \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " Jul 21 22:31:35.595600 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595427 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tls-certs\") pod \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " Jul 21 22:31:35.595600 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595452 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tmp-dir\") pod \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " Jul 21 22:31:35.595600 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595481 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-dshm\") pod \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " Jul 21 22:31:35.595600 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595535 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zr7b5\" (UniqueName: \"kubernetes.io/projected/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-kube-api-access-zr7b5\") pod \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\" (UID: \"43554f7e-f3db-4c46-9b36-25cb03b2f1d7\") " Jul 21 22:31:35.595810 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595695 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-model-cache" (OuterVolumeSpecName: "model-cache") pod "43554f7e-f3db-4c46-9b36-25cb03b2f1d7" (UID: "43554f7e-f3db-4c46-9b36-25cb03b2f1d7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:31:35.595810 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595786 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "43554f7e-f3db-4c46-9b36-25cb03b2f1d7" (UID: "43554f7e-f3db-4c46-9b36-25cb03b2f1d7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:31:35.595810 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595797 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-home" (OuterVolumeSpecName: "home") pod "43554f7e-f3db-4c46-9b36-25cb03b2f1d7" (UID: "43554f7e-f3db-4c46-9b36-25cb03b2f1d7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:31:35.595965 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.595849 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-model-cache\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:35.597833 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.597800 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-dshm" (OuterVolumeSpecName: "dshm") pod "43554f7e-f3db-4c46-9b36-25cb03b2f1d7" (UID: "43554f7e-f3db-4c46-9b36-25cb03b2f1d7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:31:35.597949 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.597881 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-kube-api-access-zr7b5" (OuterVolumeSpecName: "kube-api-access-zr7b5") pod "43554f7e-f3db-4c46-9b36-25cb03b2f1d7" (UID: "43554f7e-f3db-4c46-9b36-25cb03b2f1d7"). InnerVolumeSpecName "kube-api-access-zr7b5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:31:35.597949 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.597880 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "43554f7e-f3db-4c46-9b36-25cb03b2f1d7" (UID: "43554f7e-f3db-4c46-9b36-25cb03b2f1d7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:31:35.696538 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.696504 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-home\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:35.696538 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.696531 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tls-certs\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:35.696538 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.696542 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-tmp-dir\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:35.696538 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.696549 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-dshm\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:35.696792 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:35.696558 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zr7b5\" (UniqueName: \"kubernetes.io/projected/43554f7e-f3db-4c46-9b36-25cb03b2f1d7-kube-api-access-zr7b5\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:31:36.390434 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:36.390398 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" path="/var/lib/kubelet/pods/4cc116f5-c84f-482f-a9c4-ed62bacf6e9b/volumes" Jul 21 22:31:36.458632 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:36.458596 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" event={"ID":"43554f7e-f3db-4c46-9b36-25cb03b2f1d7","Type":"ContainerDied","Data":"444251428ce49a14fdedec0d23768f6a6811b74fc279181b3440223b699053bc"} Jul 21 22:31:36.458632 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:36.458641 2557 scope.go:117] "RemoveContainer" containerID="8855f500daded7de82c5a6b32ab78c8d3b327f25a9d7596639fd60f22b1fb5ab" Jul 21 22:31:36.458849 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:36.458606 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf" Jul 21 22:31:36.475741 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:36.475712 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf"] Jul 21 22:31:36.479643 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:36.479620 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/auth-test-model-b77603-kserve-c78c8f4f9-krgvf"] Jul 21 22:31:38.389720 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:38.389685 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43554f7e-f3db-4c46-9b36-25cb03b2f1d7" path="/var/lib/kubelet/pods/43554f7e-f3db-4c46-9b36-25cb03b2f1d7/volumes" Jul 21 22:31:39.642518 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:39.642487 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:31:58.473338 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.473303 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv"] Jul 21 22:31:58.473795 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.473625 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43554f7e-f3db-4c46-9b36-25cb03b2f1d7" containerName="main" Jul 21 22:31:58.473795 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.473635 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="43554f7e-f3db-4c46-9b36-25cb03b2f1d7" containerName="main" Jul 21 22:31:58.473795 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.473653 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" containerName="main" Jul 21 22:31:58.473795 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.473659 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" containerName="main" Jul 21 22:31:58.473795 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.473720 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="4cc116f5-c84f-482f-a9c4-ed62bacf6e9b" containerName="main" Jul 21 22:31:58.473795 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.473732 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="43554f7e-f3db-4c46-9b36-25cb03b2f1d7" containerName="main" Jul 21 22:31:58.476621 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.476603 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.479256 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.479232 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"openshift-service-ca.crt\"" Jul 21 22:31:58.480059 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.480040 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"sub-test-model-b77603-kserve-self-signed-certs\"" Jul 21 22:31:58.480330 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.480313 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"kube-root-ca.crt\"" Jul 21 22:31:58.480388 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.480366 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-b77603\"/\"default-dockercfg-p4lx2\"" Jul 21 22:31:58.485060 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.485034 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv"] Jul 21 22:31:58.586414 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.586378 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-dshm\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.586414 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.586412 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f4d03-a874-4066-8796-98c27599eb94-tls-certs\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.586620 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.586429 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-tmp-dir\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.586620 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.586478 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-model-cache\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.586620 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.586538 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-home\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.586620 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.586572 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfm97\" (UniqueName: \"kubernetes.io/projected/b19f4d03-a874-4066-8796-98c27599eb94-kube-api-access-nfm97\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.687888 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.687851 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-home\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.688067 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.687905 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfm97\" (UniqueName: \"kubernetes.io/projected/b19f4d03-a874-4066-8796-98c27599eb94-kube-api-access-nfm97\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.688067 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.687957 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-dshm\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.688067 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.687984 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f4d03-a874-4066-8796-98c27599eb94-tls-certs\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.688067 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.688008 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-tmp-dir\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.688067 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.688047 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-model-cache\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.688373 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.688279 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-home\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.688462 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.688438 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-tmp-dir\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.688534 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.688511 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-model-cache\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.690378 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.690351 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-dshm\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.690673 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.690652 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f4d03-a874-4066-8796-98c27599eb94-tls-certs\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.697841 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.697814 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfm97\" (UniqueName: \"kubernetes.io/projected/b19f4d03-a874-4066-8796-98c27599eb94-kube-api-access-nfm97\") pod \"sub-test-model-b77603-kserve-7954c57c4f-7f6nv\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.786842 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.786760 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:31:58.928198 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:58.928170 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv"] Jul 21 22:31:58.929737 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:31:58.929699 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb19f4d03_a874_4066_8796_98c27599eb94.slice/crio-49a586bd9c1d15b4e2e673490e32bf71ba68eb0cc732a6bfcd0724eace9c294b WatchSource:0}: Error finding container 49a586bd9c1d15b4e2e673490e32bf71ba68eb0cc732a6bfcd0724eace9c294b: Status 404 returned error can't find the container with id 49a586bd9c1d15b4e2e673490e32bf71ba68eb0cc732a6bfcd0724eace9c294b Jul 21 22:31:59.572670 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:59.572637 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" event={"ID":"b19f4d03-a874-4066-8796-98c27599eb94","Type":"ContainerStarted","Data":"109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061"} Jul 21 22:31:59.572670 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:59.572677 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" event={"ID":"b19f4d03-a874-4066-8796-98c27599eb94","Type":"ContainerStarted","Data":"49a586bd9c1d15b4e2e673490e32bf71ba68eb0cc732a6bfcd0724eace9c294b"} Jul 21 22:31:59.592825 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:31:59.592774 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" podStartSLOduration=1.592757073 podStartE2EDuration="1.592757073s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:31:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:31:59.591163016 +0000 UTC m=+2619.697127184" watchObservedRunningTime="2026-07-21 22:31:59.592757073 +0000 UTC m=+2619.698721226" Jul 21 22:32:02.249402 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:02.249368 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:32:08.787033 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:08.786999 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:32:08.787600 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:08.787053 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:32:08.799417 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:08.799384 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:32:09.634257 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:09.634224 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:32:21.845123 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:21.845053 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:32:41.968910 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:41.968873 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:32:48.391807 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:48.391765 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv"] Jul 21 22:32:58.535030 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:32:58.534991 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:33:00.895312 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:00.895279 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx"] Jul 21 22:33:00.898552 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:00.898535 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:00.901468 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:00.901446 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"kube-root-ca.crt\"" Jul 21 22:33:00.901577 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:00.901543 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"openshift-service-ca.crt\"" Jul 21 22:33:00.902999 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:00.902983 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"default-dockercfg-d8nnn\"" Jul 21 22:33:00.902999 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:00.902988 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-131f31\"/\"rate-test-model-131f31-kserve-self-signed-certs\"" Jul 21 22:33:00.913709 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:00.913688 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx"] Jul 21 22:33:01.033454 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.033411 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-model-cache\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.033663 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.033494 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tls-certs\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.033663 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.033527 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-dshm\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.033663 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.033566 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-home\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.033663 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.033647 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b25gz\" (UniqueName: \"kubernetes.io/projected/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-kube-api-access-b25gz\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.033872 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.033744 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tmp-dir\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135201 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135172 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b25gz\" (UniqueName: \"kubernetes.io/projected/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-kube-api-access-b25gz\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135403 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135218 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tmp-dir\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135403 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135247 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-model-cache\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135403 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135286 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tls-certs\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135403 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135304 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-dshm\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135403 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135328 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-home\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135696 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135674 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tmp-dir\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135749 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135702 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-home\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.135749 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.135719 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-model-cache\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.137812 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.137790 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-dshm\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.138014 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.137999 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tls-certs\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.145891 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.145829 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b25gz\" (UniqueName: \"kubernetes.io/projected/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-kube-api-access-b25gz\") pod \"rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.207291 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.207252 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:01.363208 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.363176 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx"] Jul 21 22:33:01.365485 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:33:01.365445 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda6dcdb0c_83b9_4da7_8a8a_807afb269b7c.slice/crio-734227eafdba18dddcc9566a7b1ff77b3a5b04b765a4be7ac60689c99851ebc2 WatchSource:0}: Error finding container 734227eafdba18dddcc9566a7b1ff77b3a5b04b765a4be7ac60689c99851ebc2: Status 404 returned error can't find the container with id 734227eafdba18dddcc9566a7b1ff77b3a5b04b765a4be7ac60689c99851ebc2 Jul 21 22:33:01.884845 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.884810 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" event={"ID":"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c","Type":"ContainerStarted","Data":"608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f"} Jul 21 22:33:01.885035 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.884856 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" event={"ID":"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c","Type":"ContainerStarted","Data":"734227eafdba18dddcc9566a7b1ff77b3a5b04b765a4be7ac60689c99851ebc2"} Jul 21 22:33:01.907723 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:01.907673 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" podStartSLOduration=1.9076561459999999 podStartE2EDuration="1.907656146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:33:01.904978522 +0000 UTC m=+2682.010942669" watchObservedRunningTime="2026-07-21 22:33:01.907656146 +0000 UTC m=+2682.013620321" Jul 21 22:33:03.411928 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.411875 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" podUID="b19f4d03-a874-4066-8796-98c27599eb94" containerName="main" containerID="cri-o://109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061" gracePeriod=45 Jul 21 22:33:03.662297 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.662237 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:33:03.756482 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756455 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nfm97\" (UniqueName: \"kubernetes.io/projected/b19f4d03-a874-4066-8796-98c27599eb94-kube-api-access-nfm97\") pod \"b19f4d03-a874-4066-8796-98c27599eb94\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " Jul 21 22:33:03.756676 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756492 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-home\") pod \"b19f4d03-a874-4066-8796-98c27599eb94\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " Jul 21 22:33:03.756676 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756514 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-tmp-dir\") pod \"b19f4d03-a874-4066-8796-98c27599eb94\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " Jul 21 22:33:03.756676 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756546 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f4d03-a874-4066-8796-98c27599eb94-tls-certs\") pod \"b19f4d03-a874-4066-8796-98c27599eb94\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " Jul 21 22:33:03.756676 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756597 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-model-cache\") pod \"b19f4d03-a874-4066-8796-98c27599eb94\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " Jul 21 22:33:03.756676 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756639 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-dshm\") pod \"b19f4d03-a874-4066-8796-98c27599eb94\" (UID: \"b19f4d03-a874-4066-8796-98c27599eb94\") " Jul 21 22:33:03.756958 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756774 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-home" (OuterVolumeSpecName: "home") pod "b19f4d03-a874-4066-8796-98c27599eb94" (UID: "b19f4d03-a874-4066-8796-98c27599eb94"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:33:03.756958 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756791 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b19f4d03-a874-4066-8796-98c27599eb94" (UID: "b19f4d03-a874-4066-8796-98c27599eb94"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:33:03.756958 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756868 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-model-cache" (OuterVolumeSpecName: "model-cache") pod "b19f4d03-a874-4066-8796-98c27599eb94" (UID: "b19f4d03-a874-4066-8796-98c27599eb94"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:33:03.756958 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756921 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-home\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:33:03.756958 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.756941 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-tmp-dir\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:33:03.758860 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.758834 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b19f4d03-a874-4066-8796-98c27599eb94-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b19f4d03-a874-4066-8796-98c27599eb94" (UID: "b19f4d03-a874-4066-8796-98c27599eb94"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:33:03.758860 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.758844 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b19f4d03-a874-4066-8796-98c27599eb94-kube-api-access-nfm97" (OuterVolumeSpecName: "kube-api-access-nfm97") pod "b19f4d03-a874-4066-8796-98c27599eb94" (UID: "b19f4d03-a874-4066-8796-98c27599eb94"). InnerVolumeSpecName "kube-api-access-nfm97". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:33:03.759005 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.758904 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-dshm" (OuterVolumeSpecName: "dshm") pod "b19f4d03-a874-4066-8796-98c27599eb94" (UID: "b19f4d03-a874-4066-8796-98c27599eb94"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:33:03.858180 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.858148 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b19f4d03-a874-4066-8796-98c27599eb94-tls-certs\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:33:03.858180 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.858176 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-model-cache\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:33:03.858180 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.858184 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b19f4d03-a874-4066-8796-98c27599eb94-dshm\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:33:03.858396 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.858194 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfm97\" (UniqueName: \"kubernetes.io/projected/b19f4d03-a874-4066-8796-98c27599eb94-kube-api-access-nfm97\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:33:03.902445 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.902411 2557 generic.go:356] "Generic (PLEG): container finished" podID="b19f4d03-a874-4066-8796-98c27599eb94" containerID="109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061" exitCode=0 Jul 21 22:33:03.902633 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.902486 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" Jul 21 22:33:03.902633 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.902502 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" event={"ID":"b19f4d03-a874-4066-8796-98c27599eb94","Type":"ContainerDied","Data":"109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061"} Jul 21 22:33:03.902633 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.902554 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv" event={"ID":"b19f4d03-a874-4066-8796-98c27599eb94","Type":"ContainerDied","Data":"49a586bd9c1d15b4e2e673490e32bf71ba68eb0cc732a6bfcd0724eace9c294b"} Jul 21 22:33:03.902633 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.902576 2557 scope.go:117] "RemoveContainer" containerID="109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061" Jul 21 22:33:03.912329 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.912287 2557 scope.go:117] "RemoveContainer" containerID="109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061" Jul 21 22:33:03.912554 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:33:03.912532 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061\": container with ID starting with 109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061 not found: ID does not exist" containerID="109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061" Jul 21 22:33:03.912621 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.912567 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061"} err="failed to get container status \"109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061\": rpc error: code = NotFound desc = could not find container \"109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061\": container with ID starting with 109eb48ee4a51f756e4a08924df3748df35157fb0b63536d92863d85374ec061 not found: ID does not exist" Jul 21 22:33:03.925962 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.925927 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv"] Jul 21 22:33:03.930623 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:03.930603 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-b77603/sub-test-model-b77603-kserve-7954c57c4f-7f6nv"] Jul 21 22:33:04.390222 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:04.390190 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b19f4d03-a874-4066-8796-98c27599eb94" path="/var/lib/kubelet/pods/b19f4d03-a874-4066-8796-98c27599eb94/volumes" Jul 21 22:33:05.435538 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:05.435506 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:33:11.208414 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:11.208321 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:11.208414 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:11.208395 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:11.221209 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:11.221172 2557 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:11.952966 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:11.952926 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:33:20.505971 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:20.505935 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:33:20.514071 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:20.514050 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:33:23.533171 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:23.533131 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:33:40.032396 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:40.032357 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:33:55.730075 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:33:55.730037 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:34:02.710465 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:02.710415 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx"] Jul 21 22:34:04.478524 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.478493 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-139937563721264"] Jul 21 22:34:04.478922 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.478832 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b19f4d03-a874-4066-8796-98c27599eb94" containerName="main" Jul 21 22:34:04.478922 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.478849 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="b19f4d03-a874-4066-8796-98c27599eb94" containerName="main" Jul 21 22:34:04.479012 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.478930 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="b19f4d03-a874-4066-8796-98c27599eb94" containerName="main" Jul 21 22:34:04.481904 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.481880 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563721264" Jul 21 22:34:04.484407 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.484382 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-vr99r\"" Jul 21 22:34:04.489818 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.489795 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563721264"] Jul 21 22:34:04.581660 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.581610 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kthf\" (UniqueName: \"kubernetes.io/projected/c290d64c-f1fc-4223-9cba-1576f2b7aa1f-kube-api-access-8kthf\") pod \"test-curl-7616-139937563721264\" (UID: \"c290d64c-f1fc-4223-9cba-1576f2b7aa1f\") " pod="opendatahub/test-curl-7616-139937563721264" Jul 21 22:34:04.682636 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.682598 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kthf\" (UniqueName: \"kubernetes.io/projected/c290d64c-f1fc-4223-9cba-1576f2b7aa1f-kube-api-access-8kthf\") pod \"test-curl-7616-139937563721264\" (UID: \"c290d64c-f1fc-4223-9cba-1576f2b7aa1f\") " pod="opendatahub/test-curl-7616-139937563721264" Jul 21 22:34:04.694029 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.694007 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kthf\" (UniqueName: \"kubernetes.io/projected/c290d64c-f1fc-4223-9cba-1576f2b7aa1f-kube-api-access-8kthf\") pod \"test-curl-7616-139937563721264\" (UID: \"c290d64c-f1fc-4223-9cba-1576f2b7aa1f\") " pod="opendatahub/test-curl-7616-139937563721264" Jul 21 22:34:04.792454 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.792368 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563721264" Jul 21 22:34:04.928856 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.928828 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563721264"] Jul 21 22:34:04.930017 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:34:04.929988 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc290d64c_f1fc_4223_9cba_1576f2b7aa1f.slice/crio-4e6b5358e4784cd2dc7e94101eb46984bd1c169df95eeade6ff6a915ff99b34f WatchSource:0}: Error finding container 4e6b5358e4784cd2dc7e94101eb46984bd1c169df95eeade6ff6a915ff99b34f: Status 404 returned error can't find the container with id 4e6b5358e4784cd2dc7e94101eb46984bd1c169df95eeade6ff6a915ff99b34f Jul 21 22:34:04.931454 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:04.931438 2557 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 21 22:34:05.208753 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:05.208664 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563721264" event={"ID":"c290d64c-f1fc-4223-9cba-1576f2b7aa1f","Type":"ContainerStarted","Data":"4e6b5358e4784cd2dc7e94101eb46984bd1c169df95eeade6ff6a915ff99b34f"} Jul 21 22:34:06.214989 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:06.214956 2557 generic.go:356] "Generic (PLEG): container finished" podID="c290d64c-f1fc-4223-9cba-1576f2b7aa1f" containerID="a2b0ce51662e8ecac1e9d5d6a31789fb0bd9e76cd578241028b5f36b90be4fbf" exitCode=0 Jul 21 22:34:06.215390 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:06.215039 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563721264" event={"ID":"c290d64c-f1fc-4223-9cba-1576f2b7aa1f","Type":"ContainerDied","Data":"a2b0ce51662e8ecac1e9d5d6a31789fb0bd9e76cd578241028b5f36b90be4fbf"} Jul 21 22:34:07.351517 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.351499 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563721264" Jul 21 22:34:07.392009 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.391984 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7616-139937563721264_c290d64c-f1fc-4223-9cba-1576f2b7aa1f/test-curl-7616-139937563721264/0.log" Jul 21 22:34:07.405283 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.405255 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8kthf\" (UniqueName: \"kubernetes.io/projected/c290d64c-f1fc-4223-9cba-1576f2b7aa1f-kube-api-access-8kthf\") pod \"c290d64c-f1fc-4223-9cba-1576f2b7aa1f\" (UID: \"c290d64c-f1fc-4223-9cba-1576f2b7aa1f\") " Jul 21 22:34:07.407482 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.407455 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c290d64c-f1fc-4223-9cba-1576f2b7aa1f-kube-api-access-8kthf" (OuterVolumeSpecName: "kube-api-access-8kthf") pod "c290d64c-f1fc-4223-9cba-1576f2b7aa1f" (UID: "c290d64c-f1fc-4223-9cba-1576f2b7aa1f"). InnerVolumeSpecName "kube-api-access-8kthf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:07.416575 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.416551 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-139937563721264"] Jul 21 22:34:07.418624 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.418602 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-139937563721264"] Jul 21 22:34:07.506536 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.506496 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kthf\" (UniqueName: \"kubernetes.io/projected/c290d64c-f1fc-4223-9cba-1576f2b7aa1f-kube-api-access-8kthf\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:07.546825 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.546790 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-139937563721520"] Jul 21 22:34:07.547147 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.547133 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c290d64c-f1fc-4223-9cba-1576f2b7aa1f" containerName="test-curl-7616-139937563721264" Jul 21 22:34:07.547203 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.547148 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="c290d64c-f1fc-4223-9cba-1576f2b7aa1f" containerName="test-curl-7616-139937563721264" Jul 21 22:34:07.547260 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.547226 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="c290d64c-f1fc-4223-9cba-1576f2b7aa1f" containerName="test-curl-7616-139937563721264" Jul 21 22:34:07.550231 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.550214 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:07.558982 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.558957 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563721520"] Jul 21 22:34:07.607864 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.607826 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2jx\" (UniqueName: \"kubernetes.io/projected/ae554611-57d1-460f-82f6-02d91efe87b3-kube-api-access-wr2jx\") pod \"test-curl-7616-139937563721520\" (UID: \"ae554611-57d1-460f-82f6-02d91efe87b3\") " pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:07.708798 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.708750 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2jx\" (UniqueName: \"kubernetes.io/projected/ae554611-57d1-460f-82f6-02d91efe87b3-kube-api-access-wr2jx\") pod \"test-curl-7616-139937563721520\" (UID: \"ae554611-57d1-460f-82f6-02d91efe87b3\") " pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:07.718763 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.718737 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2jx\" (UniqueName: \"kubernetes.io/projected/ae554611-57d1-460f-82f6-02d91efe87b3-kube-api-access-wr2jx\") pod \"test-curl-7616-139937563721520\" (UID: \"ae554611-57d1-460f-82f6-02d91efe87b3\") " pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:07.859030 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:07.858952 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:08.003577 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:08.003466 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563721520"] Jul 21 22:34:08.006060 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:34:08.006032 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae554611_57d1_460f_82f6_02d91efe87b3.slice/crio-bb54b792cb6a20d74ba40e02649ee63f2e8ce66bd7f4555ad32187b13fd57fbc WatchSource:0}: Error finding container bb54b792cb6a20d74ba40e02649ee63f2e8ce66bd7f4555ad32187b13fd57fbc: Status 404 returned error can't find the container with id bb54b792cb6a20d74ba40e02649ee63f2e8ce66bd7f4555ad32187b13fd57fbc Jul 21 22:34:08.226122 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:08.226075 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563721264" Jul 21 22:34:08.226319 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:08.226143 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e6b5358e4784cd2dc7e94101eb46984bd1c169df95eeade6ff6a915ff99b34f" Jul 21 22:34:08.227350 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:08.227327 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563721520" event={"ID":"ae554611-57d1-460f-82f6-02d91efe87b3","Type":"ContainerStarted","Data":"bb54b792cb6a20d74ba40e02649ee63f2e8ce66bd7f4555ad32187b13fd57fbc"} Jul 21 22:34:08.236763 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:08.236735 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563721264\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="c290d64c-f1fc-4223-9cba-1576f2b7aa1f" pod="opendatahub/test-curl-7616-139937563721264" Jul 21 22:34:08.399717 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:08.399682 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c290d64c-f1fc-4223-9cba-1576f2b7aa1f" path="/var/lib/kubelet/pods/c290d64c-f1fc-4223-9cba-1576f2b7aa1f/volumes" Jul 21 22:34:09.234561 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:09.234528 2557 generic.go:356] "Generic (PLEG): container finished" podID="ae554611-57d1-460f-82f6-02d91efe87b3" containerID="c54526cf03974eab7be16bec35451b15e6232388700c0a805c2b8f536f1c38a5" exitCode=0 Jul 21 22:34:09.234766 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:09.234621 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563721520" event={"ID":"ae554611-57d1-460f-82f6-02d91efe87b3","Type":"ContainerDied","Data":"c54526cf03974eab7be16bec35451b15e6232388700c0a805c2b8f536f1c38a5"} Jul 21 22:34:09.935282 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:09.935243 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:34:10.373634 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.373607 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:10.396838 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.396811 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7616-139937563721520_ae554611-57d1-460f-82f6-02d91efe87b3/test-curl-7616-139937563721520/0.log" Jul 21 22:34:10.420667 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.420640 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-139937563721520"] Jul 21 22:34:10.424528 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.424499 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-139937563721520"] Jul 21 22:34:10.433039 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.433018 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr2jx\" (UniqueName: \"kubernetes.io/projected/ae554611-57d1-460f-82f6-02d91efe87b3-kube-api-access-wr2jx\") pod \"ae554611-57d1-460f-82f6-02d91efe87b3\" (UID: \"ae554611-57d1-460f-82f6-02d91efe87b3\") " Jul 21 22:34:10.435224 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.435200 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae554611-57d1-460f-82f6-02d91efe87b3-kube-api-access-wr2jx" (OuterVolumeSpecName: "kube-api-access-wr2jx") pod "ae554611-57d1-460f-82f6-02d91efe87b3" (UID: "ae554611-57d1-460f-82f6-02d91efe87b3"). InnerVolumeSpecName "kube-api-access-wr2jx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:10.533863 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.533777 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2jx\" (UniqueName: \"kubernetes.io/projected/ae554611-57d1-460f-82f6-02d91efe87b3-kube-api-access-wr2jx\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:10.588340 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.588299 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:10.588647 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.588634 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae554611-57d1-460f-82f6-02d91efe87b3" containerName="test-curl-7616-139937563721520" Jul 21 22:34:10.588647 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.588648 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae554611-57d1-460f-82f6-02d91efe87b3" containerName="test-curl-7616-139937563721520" Jul 21 22:34:10.588735 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.588719 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae554611-57d1-460f-82f6-02d91efe87b3" containerName="test-curl-7616-139937563721520" Jul 21 22:34:10.591626 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.591601 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:10.599124 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.599078 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:10.635196 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.635159 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8wn4\" (UniqueName: \"kubernetes.io/projected/f3a625dd-fd02-46cf-ae18-8d2eb667a7ba-kube-api-access-v8wn4\") pod \"test-curl-7616-139937563644960\" (UID: \"f3a625dd-fd02-46cf-ae18-8d2eb667a7ba\") " pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:10.736485 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.736447 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8wn4\" (UniqueName: \"kubernetes.io/projected/f3a625dd-fd02-46cf-ae18-8d2eb667a7ba-kube-api-access-v8wn4\") pod \"test-curl-7616-139937563644960\" (UID: \"f3a625dd-fd02-46cf-ae18-8d2eb667a7ba\") " pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:10.747421 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.747386 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8wn4\" (UniqueName: \"kubernetes.io/projected/f3a625dd-fd02-46cf-ae18-8d2eb667a7ba-kube-api-access-v8wn4\") pod \"test-curl-7616-139937563644960\" (UID: \"f3a625dd-fd02-46cf-ae18-8d2eb667a7ba\") " pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:10.903065 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:10.902965 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:11.036260 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:11.036236 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:11.038138 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:34:11.038078 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf3a625dd_fd02_46cf_ae18_8d2eb667a7ba.slice/crio-60440344847fa87558c46900082f9e4a692d8aff3cf3748cecd56e12e666b3b2 WatchSource:0}: Error finding container 60440344847fa87558c46900082f9e4a692d8aff3cf3748cecd56e12e666b3b2: Status 404 returned error can't find the container with id 60440344847fa87558c46900082f9e4a692d8aff3cf3748cecd56e12e666b3b2 Jul 21 22:34:11.246025 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:11.245994 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:11.246285 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:11.246036 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb54b792cb6a20d74ba40e02649ee63f2e8ce66bd7f4555ad32187b13fd57fbc" Jul 21 22:34:11.247184 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:11.247152 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563644960" event={"ID":"f3a625dd-fd02-46cf-ae18-8d2eb667a7ba","Type":"ContainerStarted","Data":"60440344847fa87558c46900082f9e4a692d8aff3cf3748cecd56e12e666b3b2"} Jul 21 22:34:11.256311 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:11.256288 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563721520\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="ae554611-57d1-460f-82f6-02d91efe87b3" pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:12.253754 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:12.253719 2557 generic.go:356] "Generic (PLEG): container finished" podID="f3a625dd-fd02-46cf-ae18-8d2eb667a7ba" containerID="0366a1f50e04e1910b6133688697993ed3788812274037323f86d2806f23d7c6" exitCode=0 Jul 21 22:34:12.254221 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:12.253806 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563644960" event={"ID":"f3a625dd-fd02-46cf-ae18-8d2eb667a7ba","Type":"ContainerDied","Data":"0366a1f50e04e1910b6133688697993ed3788812274037323f86d2806f23d7c6"} Jul 21 22:34:12.255949 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:12.255921 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563721520\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="ae554611-57d1-460f-82f6-02d91efe87b3" pod="opendatahub/test-curl-7616-139937563721520" Jul 21 22:34:12.395026 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:12.392015 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae554611-57d1-460f-82f6-02d91efe87b3" path="/var/lib/kubelet/pods/ae554611-57d1-460f-82f6-02d91efe87b3/volumes" Jul 21 22:34:13.392038 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.392013 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:13.413625 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.413595 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7616-139937563644960_f3a625dd-fd02-46cf-ae18-8d2eb667a7ba/test-curl-7616-139937563644960/0.log" Jul 21 22:34:13.437935 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.437905 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:13.439978 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.439956 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:13.457722 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.457698 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v8wn4\" (UniqueName: \"kubernetes.io/projected/f3a625dd-fd02-46cf-ae18-8d2eb667a7ba-kube-api-access-v8wn4\") pod \"f3a625dd-fd02-46cf-ae18-8d2eb667a7ba\" (UID: \"f3a625dd-fd02-46cf-ae18-8d2eb667a7ba\") " Jul 21 22:34:13.459836 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.459814 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3a625dd-fd02-46cf-ae18-8d2eb667a7ba-kube-api-access-v8wn4" (OuterVolumeSpecName: "kube-api-access-v8wn4") pod "f3a625dd-fd02-46cf-ae18-8d2eb667a7ba" (UID: "f3a625dd-fd02-46cf-ae18-8d2eb667a7ba"). InnerVolumeSpecName "kube-api-access-v8wn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:13.558547 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.558459 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8wn4\" (UniqueName: \"kubernetes.io/projected/f3a625dd-fd02-46cf-ae18-8d2eb667a7ba-kube-api-access-v8wn4\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:13.562405 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.562374 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:13.562713 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.562700 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3a625dd-fd02-46cf-ae18-8d2eb667a7ba" containerName="test-curl-7616-139937563644960" Jul 21 22:34:13.562774 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.562717 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3a625dd-fd02-46cf-ae18-8d2eb667a7ba" containerName="test-curl-7616-139937563644960" Jul 21 22:34:13.562816 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.562781 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3a625dd-fd02-46cf-ae18-8d2eb667a7ba" containerName="test-curl-7616-139937563644960" Jul 21 22:34:13.565854 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.565838 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:13.577526 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.577505 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:13.658835 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.658807 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4478\" (UniqueName: \"kubernetes.io/projected/5d862c7e-dae0-4532-86ce-1b4a61e9c971-kube-api-access-d4478\") pod \"test-curl-7616-139937563644960\" (UID: \"5d862c7e-dae0-4532-86ce-1b4a61e9c971\") " pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:13.759588 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.759552 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4478\" (UniqueName: \"kubernetes.io/projected/5d862c7e-dae0-4532-86ce-1b4a61e9c971-kube-api-access-d4478\") pod \"test-curl-7616-139937563644960\" (UID: \"5d862c7e-dae0-4532-86ce-1b4a61e9c971\") " pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:13.768605 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.768577 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4478\" (UniqueName: \"kubernetes.io/projected/5d862c7e-dae0-4532-86ce-1b4a61e9c971-kube-api-access-d4478\") pod \"test-curl-7616-139937563644960\" (UID: \"5d862c7e-dae0-4532-86ce-1b4a61e9c971\") " pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:13.874893 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:13.874800 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:14.009466 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:14.009436 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:14.011075 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:34:14.011049 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5d862c7e_dae0_4532_86ce_1b4a61e9c971.slice/crio-6e88711bc3e3d07646a1d0b1dbd888ad98a057f38752e81ccd154ed512798198 WatchSource:0}: Error finding container 6e88711bc3e3d07646a1d0b1dbd888ad98a057f38752e81ccd154ed512798198: Status 404 returned error can't find the container with id 6e88711bc3e3d07646a1d0b1dbd888ad98a057f38752e81ccd154ed512798198 Jul 21 22:34:14.265107 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:14.265054 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:14.265321 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:14.265054 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="60440344847fa87558c46900082f9e4a692d8aff3cf3748cecd56e12e666b3b2" Jul 21 22:34:14.266347 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:14.266318 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563644960" event={"ID":"5d862c7e-dae0-4532-86ce-1b4a61e9c971","Type":"ContainerStarted","Data":"6e88711bc3e3d07646a1d0b1dbd888ad98a057f38752e81ccd154ed512798198"} Jul 21 22:34:14.276710 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:14.276679 2557 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7616-139937563644960" oldPodUID="f3a625dd-fd02-46cf-ae18-8d2eb667a7ba" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" Jul 21 22:34:14.390242 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:14.390205 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3a625dd-fd02-46cf-ae18-8d2eb667a7ba" path="/var/lib/kubelet/pods/f3a625dd-fd02-46cf-ae18-8d2eb667a7ba/volumes" Jul 21 22:34:15.273357 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:15.273310 2557 generic.go:356] "Generic (PLEG): container finished" podID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" containerID="a36a7913f0b31a6d7c55e573110ddc312c296ea4f7352661b13b80f7a7c58fa7" exitCode=0 Jul 21 22:34:15.273738 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:15.273397 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563644960" event={"ID":"5d862c7e-dae0-4532-86ce-1b4a61e9c971","Type":"ContainerDied","Data":"a36a7913f0b31a6d7c55e573110ddc312c296ea4f7352661b13b80f7a7c58fa7"} Jul 21 22:34:16.412320 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.412294 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:16.433263 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.433230 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7616-139937563644960_5d862c7e-dae0-4532-86ce-1b4a61e9c971/test-curl-7616-139937563644960/0.log" Jul 21 22:34:16.457804 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.457777 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:16.465619 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.465593 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-139937563644960"] Jul 21 22:34:16.483684 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.483661 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4478\" (UniqueName: \"kubernetes.io/projected/5d862c7e-dae0-4532-86ce-1b4a61e9c971-kube-api-access-d4478\") pod \"5d862c7e-dae0-4532-86ce-1b4a61e9c971\" (UID: \"5d862c7e-dae0-4532-86ce-1b4a61e9c971\") " Jul 21 22:34:16.485849 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.485826 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d862c7e-dae0-4532-86ce-1b4a61e9c971-kube-api-access-d4478" (OuterVolumeSpecName: "kube-api-access-d4478") pod "5d862c7e-dae0-4532-86ce-1b4a61e9c971" (UID: "5d862c7e-dae0-4532-86ce-1b4a61e9c971"). InnerVolumeSpecName "kube-api-access-d4478". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:16.585070 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.584980 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4478\" (UniqueName: \"kubernetes.io/projected/5d862c7e-dae0-4532-86ce-1b4a61e9c971-kube-api-access-d4478\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:16.610033 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.610002 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:16.610358 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.610345 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" containerName="test-curl-7616-139937563644960" Jul 21 22:34:16.610411 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.610361 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" containerName="test-curl-7616-139937563644960" Jul 21 22:34:16.610449 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.610421 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" containerName="test-curl-7616-139937563644960" Jul 21 22:34:16.613365 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.613347 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:16.621124 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.621074 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:16.686319 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.686290 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkxk8\" (UniqueName: \"kubernetes.io/projected/9c0dd154-1d09-4025-be2c-1624adf2e0f2-kube-api-access-jkxk8\") pod \"test-curl-7616-139937563645968\" (UID: \"9c0dd154-1d09-4025-be2c-1624adf2e0f2\") " pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:16.787690 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.787646 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkxk8\" (UniqueName: \"kubernetes.io/projected/9c0dd154-1d09-4025-be2c-1624adf2e0f2-kube-api-access-jkxk8\") pod \"test-curl-7616-139937563645968\" (UID: \"9c0dd154-1d09-4025-be2c-1624adf2e0f2\") " pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:16.796808 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.796776 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkxk8\" (UniqueName: \"kubernetes.io/projected/9c0dd154-1d09-4025-be2c-1624adf2e0f2-kube-api-access-jkxk8\") pod \"test-curl-7616-139937563645968\" (UID: \"9c0dd154-1d09-4025-be2c-1624adf2e0f2\") " pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:16.923164 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:16.923047 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:17.077181 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:17.077154 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:17.079373 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:34:17.079333 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9c0dd154_1d09_4025_be2c_1624adf2e0f2.slice/crio-4793725e8f6a4ab682ada86c4339e833732d7f3db3fc2ed96fccdd9c770c0dd6 WatchSource:0}: Error finding container 4793725e8f6a4ab682ada86c4339e833732d7f3db3fc2ed96fccdd9c770c0dd6: Status 404 returned error can't find the container with id 4793725e8f6a4ab682ada86c4339e833732d7f3db3fc2ed96fccdd9c770c0dd6 Jul 21 22:34:17.285510 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:17.285465 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563645968" event={"ID":"9c0dd154-1d09-4025-be2c-1624adf2e0f2","Type":"ContainerStarted","Data":"4793725e8f6a4ab682ada86c4339e833732d7f3db3fc2ed96fccdd9c770c0dd6"} Jul 21 22:34:17.286762 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:17.286738 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e88711bc3e3d07646a1d0b1dbd888ad98a057f38752e81ccd154ed512798198" Jul 21 22:34:17.286762 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:17.286748 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:17.297169 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:17.297137 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563644960\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:17.730724 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:17.730659 2557 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" podUID="a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" containerName="main" containerID="cri-o://608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f" gracePeriod=45 Jul 21 22:34:17.979102 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:17.979060 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:34:17.981600 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:17.981544 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563644960\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:18.098945 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.098908 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tls-certs\") pod \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " Jul 21 22:34:18.099146 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.098969 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b25gz\" (UniqueName: \"kubernetes.io/projected/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-kube-api-access-b25gz\") pod \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " Jul 21 22:34:18.099146 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099045 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-model-cache\") pod \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " Jul 21 22:34:18.099146 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099068 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-dshm\") pod \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " Jul 21 22:34:18.099317 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099199 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-home\") pod \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " Jul 21 22:34:18.099317 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099271 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tmp-dir\") pod \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\" (UID: \"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c\") " Jul 21 22:34:18.099405 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099304 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" (UID: "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:18.099452 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099432 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-home" (OuterVolumeSpecName: "home") pod "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" (UID: "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:18.099661 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099563 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" (UID: "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:18.099661 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099608 2557 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-model-cache\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.099661 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.099625 2557 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-home\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.101227 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.101206 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" (UID: "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 21 22:34:18.101640 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.101623 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-dshm" (OuterVolumeSpecName: "dshm") pod "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" (UID: "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 21 22:34:18.101729 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.101644 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-kube-api-access-b25gz" (OuterVolumeSpecName: "kube-api-access-b25gz") pod "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" (UID: "a6dcdb0c-83b9-4da7-8a8a-807afb269b7c"). InnerVolumeSpecName "kube-api-access-b25gz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:18.200184 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.200149 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b25gz\" (UniqueName: \"kubernetes.io/projected/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-kube-api-access-b25gz\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.200184 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.200183 2557 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-dshm\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.200373 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.200198 2557 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tmp-dir\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.200373 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.200211 2557 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c-tls-certs\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:18.293116 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.293006 2557 generic.go:356] "Generic (PLEG): container finished" podID="a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" containerID="608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f" exitCode=0 Jul 21 22:34:18.293116 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.293048 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" Jul 21 22:34:18.293347 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.293129 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" event={"ID":"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c","Type":"ContainerDied","Data":"608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f"} Jul 21 22:34:18.293347 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.293176 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx" event={"ID":"a6dcdb0c-83b9-4da7-8a8a-807afb269b7c","Type":"ContainerDied","Data":"734227eafdba18dddcc9566a7b1ff77b3a5b04b765a4be7ac60689c99851ebc2"} Jul 21 22:34:18.293347 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.293200 2557 scope.go:117] "RemoveContainer" containerID="608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f" Jul 21 22:34:18.294858 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.294832 2557 generic.go:356] "Generic (PLEG): container finished" podID="9c0dd154-1d09-4025-be2c-1624adf2e0f2" containerID="25473b59e13d654f686e2a203ae9226fdaf394b3ec827dac28d21fe01e949369" exitCode=0 Jul 21 22:34:18.294964 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.294868 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563645968" event={"ID":"9c0dd154-1d09-4025-be2c-1624adf2e0f2","Type":"ContainerDied","Data":"25473b59e13d654f686e2a203ae9226fdaf394b3ec827dac28d21fe01e949369"} Jul 21 22:34:18.295781 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:18.295758 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563644960\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:18.300934 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:18.300905 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563644960\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:18.302607 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.302587 2557 scope.go:117] "RemoveContainer" containerID="608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f" Jul 21 22:34:18.302845 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:18.302826 2557 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f\": container with ID starting with 608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f not found: ID does not exist" containerID="608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f" Jul 21 22:34:18.302903 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.302854 2557 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f"} err="failed to get container status \"608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f\": rpc error: code = NotFound desc = could not find container \"608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f\": container with ID starting with 608ec3a9d08672339d4e456db1e2b21b48b8cbdf8b507bc202c43a9e8a6f351f not found: ID does not exist" Jul 21 22:34:18.318761 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:18.318730 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563644960\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" pod="opendatahub/test-curl-7616-139937563644960" Jul 21 22:34:18.332422 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.332394 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx"] Jul 21 22:34:18.337044 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.337022 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-131f31/rate-test-model-131f31-kserve-6cd5f4c7ff-p9ptx"] Jul 21 22:34:18.389959 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.389932 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d862c7e-dae0-4532-86ce-1b4a61e9c971" path="/var/lib/kubelet/pods/5d862c7e-dae0-4532-86ce-1b4a61e9c971/volumes" Jul 21 22:34:18.390340 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:18.390327 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" path="/var/lib/kubelet/pods/a6dcdb0c-83b9-4da7-8a8a-807afb269b7c/volumes" Jul 21 22:34:19.435908 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.435881 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:19.459966 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.459943 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7616-139937563645968_9c0dd154-1d09-4025-be2c-1624adf2e0f2/test-curl-7616-139937563645968/0.log" Jul 21 22:34:19.487908 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.487880 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:19.494680 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.494656 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:19.509348 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.509329 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jkxk8\" (UniqueName: \"kubernetes.io/projected/9c0dd154-1d09-4025-be2c-1624adf2e0f2-kube-api-access-jkxk8\") pod \"9c0dd154-1d09-4025-be2c-1624adf2e0f2\" (UID: \"9c0dd154-1d09-4025-be2c-1624adf2e0f2\") " Jul 21 22:34:19.511443 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.511417 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c0dd154-1d09-4025-be2c-1624adf2e0f2-kube-api-access-jkxk8" (OuterVolumeSpecName: "kube-api-access-jkxk8") pod "9c0dd154-1d09-4025-be2c-1624adf2e0f2" (UID: "9c0dd154-1d09-4025-be2c-1624adf2e0f2"). InnerVolumeSpecName "kube-api-access-jkxk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:19.610409 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.610330 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkxk8\" (UniqueName: \"kubernetes.io/projected/9c0dd154-1d09-4025-be2c-1624adf2e0f2-kube-api-access-jkxk8\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:19.621948 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.621920 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:19.622261 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.622249 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" containerName="main" Jul 21 22:34:19.622319 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.622263 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" containerName="main" Jul 21 22:34:19.622319 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.622282 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c0dd154-1d09-4025-be2c-1624adf2e0f2" containerName="test-curl-7616-139937563645968" Jul 21 22:34:19.622319 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.622287 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c0dd154-1d09-4025-be2c-1624adf2e0f2" containerName="test-curl-7616-139937563645968" Jul 21 22:34:19.622413 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.622345 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c0dd154-1d09-4025-be2c-1624adf2e0f2" containerName="test-curl-7616-139937563645968" Jul 21 22:34:19.622413 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.622354 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6dcdb0c-83b9-4da7-8a8a-807afb269b7c" containerName="main" Jul 21 22:34:19.625293 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.625276 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:19.632941 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.632919 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:19.710943 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.710907 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ztbl\" (UniqueName: \"kubernetes.io/projected/24c95235-0a91-4aa1-8b34-b49f53b1e1c6-kube-api-access-2ztbl\") pod \"test-curl-7616-139937563645968\" (UID: \"24c95235-0a91-4aa1-8b34-b49f53b1e1c6\") " pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:19.812398 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.812360 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ztbl\" (UniqueName: \"kubernetes.io/projected/24c95235-0a91-4aa1-8b34-b49f53b1e1c6-kube-api-access-2ztbl\") pod \"test-curl-7616-139937563645968\" (UID: \"24c95235-0a91-4aa1-8b34-b49f53b1e1c6\") " pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:19.821981 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.821949 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ztbl\" (UniqueName: \"kubernetes.io/projected/24c95235-0a91-4aa1-8b34-b49f53b1e1c6-kube-api-access-2ztbl\") pod \"test-curl-7616-139937563645968\" (UID: \"24c95235-0a91-4aa1-8b34-b49f53b1e1c6\") " pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:19.935065 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:19.934981 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:20.073960 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:20.073936 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:20.075721 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:34:20.075699 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod24c95235_0a91_4aa1_8b34_b49f53b1e1c6.slice/crio-069ac49d2661176330520348e6019533d12326a4c1d06d689a6b315ada9da9aa WatchSource:0}: Error finding container 069ac49d2661176330520348e6019533d12326a4c1d06d689a6b315ada9da9aa: Status 404 returned error can't find the container with id 069ac49d2661176330520348e6019533d12326a4c1d06d689a6b315ada9da9aa Jul 21 22:34:20.307702 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:20.307664 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563645968" event={"ID":"24c95235-0a91-4aa1-8b34-b49f53b1e1c6","Type":"ContainerStarted","Data":"069ac49d2661176330520348e6019533d12326a4c1d06d689a6b315ada9da9aa"} Jul 21 22:34:20.308953 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:20.308931 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:20.309079 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:20.308941 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4793725e8f6a4ab682ada86c4339e833732d7f3db3fc2ed96fccdd9c770c0dd6" Jul 21 22:34:20.320850 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:20.320826 2557 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7616-139937563645968" oldPodUID="9c0dd154-1d09-4025-be2c-1624adf2e0f2" podUID="24c95235-0a91-4aa1-8b34-b49f53b1e1c6" Jul 21 22:34:20.391707 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:20.391672 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c0dd154-1d09-4025-be2c-1624adf2e0f2" path="/var/lib/kubelet/pods/9c0dd154-1d09-4025-be2c-1624adf2e0f2/volumes" Jul 21 22:34:21.315492 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:21.315455 2557 generic.go:356] "Generic (PLEG): container finished" podID="24c95235-0a91-4aa1-8b34-b49f53b1e1c6" containerID="199fecf7a04bc49348705e0afa71002254577d3bfb90b8047709b376282b0705" exitCode=0 Jul 21 22:34:21.315843 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:21.315535 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7616-139937563645968" event={"ID":"24c95235-0a91-4aa1-8b34-b49f53b1e1c6","Type":"ContainerDied","Data":"199fecf7a04bc49348705e0afa71002254577d3bfb90b8047709b376282b0705"} Jul 21 22:34:22.455259 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:22.455231 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:22.480429 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:22.480405 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7616-139937563645968_24c95235-0a91-4aa1-8b34-b49f53b1e1c6/test-curl-7616-139937563645968/0.log" Jul 21 22:34:22.508665 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:22.508635 2557 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:22.512299 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:22.512274 2557 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7616-139937563645968"] Jul 21 22:34:22.536980 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:22.536951 2557 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2ztbl\" (UniqueName: \"kubernetes.io/projected/24c95235-0a91-4aa1-8b34-b49f53b1e1c6-kube-api-access-2ztbl\") pod \"24c95235-0a91-4aa1-8b34-b49f53b1e1c6\" (UID: \"24c95235-0a91-4aa1-8b34-b49f53b1e1c6\") " Jul 21 22:34:22.539120 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:22.539075 2557 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24c95235-0a91-4aa1-8b34-b49f53b1e1c6-kube-api-access-2ztbl" (OuterVolumeSpecName: "kube-api-access-2ztbl") pod "24c95235-0a91-4aa1-8b34-b49f53b1e1c6" (UID: "24c95235-0a91-4aa1-8b34-b49f53b1e1c6"). InnerVolumeSpecName "kube-api-access-2ztbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 21 22:34:22.638537 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:22.638454 2557 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ztbl\" (UniqueName: \"kubernetes.io/projected/24c95235-0a91-4aa1-8b34-b49f53b1e1c6-kube-api-access-2ztbl\") on node \"ip-10-0-138-36.ec2.internal\" DevicePath \"\"" Jul 21 22:34:23.327351 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:23.327324 2557 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:23.327351 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:23.327343 2557 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="069ac49d2661176330520348e6019533d12326a4c1d06d689a6b315ada9da9aa" Jul 21 22:34:23.337271 ip-10-0-138-36 kubenswrapper[2557]: E0721 22:34:23.337232 2557 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7616-139937563645968\" is forbidden: User \"system:node:ip-10-0-138-36.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-36.ec2.internal' and this object" podUID="24c95235-0a91-4aa1-8b34-b49f53b1e1c6" pod="opendatahub/test-curl-7616-139937563645968" Jul 21 22:34:24.389668 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:24.389633 2557 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24c95235-0a91-4aa1-8b34-b49f53b1e1c6" path="/var/lib/kubelet/pods/24c95235-0a91-4aa1-8b34-b49f53b1e1c6/volumes" Jul 21 22:34:36.131121 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:36.131068 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:34:58.428959 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:34:58.428923 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-lvrlh"] Jul 21 22:35:07.754749 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:07.754718 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6fd888dcb8-zdcgs_defe2a6f-3e8b-4636-9549-1c4e1376c572/authorino/0.log" Jul 21 22:35:13.589127 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:13.589070 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zr6gw_14f720e4-99c1-4618-8be1-28a4f2881372/server/0.log" Jul 21 22:35:13.706101 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:13.706068 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n4qw9_47ef4f44-ca7b-4104-9041-143b18022eb1/manager/0.log" Jul 21 22:35:13.822108 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:13.822060 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-5sfcn_63c8eb32-068c-420d-82ca-2a4cbe3cb4ab/manager/0.log" Jul 21 22:35:13.836866 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:13.836844 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-5sfcn_63c8eb32-068c-420d-82ca-2a4cbe3cb4ab/copy-manifests/0.log" Jul 21 22:35:15.717873 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.717841 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k_10c555e8-5f68-4a6c-bae0-4223d26350d7/util/0.log" Jul 21 22:35:15.726747 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.726726 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k_10c555e8-5f68-4a6c-bae0-4223d26350d7/pull/0.log" Jul 21 22:35:15.734068 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.734046 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k_10c555e8-5f68-4a6c-bae0-4223d26350d7/extract/0.log" Jul 21 22:35:15.843644 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.843609 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7_54dda3dd-b498-450e-880d-89c685642624/util/0.log" Jul 21 22:35:15.850427 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.850394 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7_54dda3dd-b498-450e-880d-89c685642624/pull/0.log" Jul 21 22:35:15.857213 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.857196 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7_54dda3dd-b498-450e-880d-89c685642624/extract/0.log" Jul 21 22:35:15.972764 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.972734 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p_3da4e59f-0739-4ae2-a15d-8f08843a165e/util/0.log" Jul 21 22:35:15.979440 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.979420 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p_3da4e59f-0739-4ae2-a15d-8f08843a165e/pull/0.log" Jul 21 22:35:15.986664 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:15.986645 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p_3da4e59f-0739-4ae2-a15d-8f08843a165e/extract/0.log" Jul 21 22:35:16.100968 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:16.100935 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245_5c647702-75d8-419b-9d6b-2b6ba74f5080/extract/0.log" Jul 21 22:35:16.107708 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:16.107686 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245_5c647702-75d8-419b-9d6b-2b6ba74f5080/util/0.log" Jul 21 22:35:16.114058 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:16.114040 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245_5c647702-75d8-419b-9d6b-2b6ba74f5080/pull/0.log" Jul 21 22:35:16.231696 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:16.231621 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6fd888dcb8-zdcgs_defe2a6f-3e8b-4636-9549-1c4e1376c572/authorino/0.log" Jul 21 22:35:16.586210 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:16.586136 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jr5pd_cf7bcd1e-4217-4651-8dfb-c33468138baa/kuadrant-console-plugin/0.log" Jul 21 22:35:16.958423 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:16.958351 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lvrlh_fada2524-786d-4b4f-a0e9-81fe443006ad/limitador/0.log" Jul 21 22:35:17.813749 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:17.813722 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594787b58-x8sdj_50b67417-ddfd-4dd5-bede-d132bc7752f2/kube-auth-proxy/0.log" Jul 21 22:35:18.313272 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:18.313240 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8649769886-nbx92_37e21193-957c-4af6-b708-b77445d6229a/router/0.log" Jul 21 22:35:19.032541 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:19.032513 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f6b99h_ac6aae96-b948-4ae8-8510-2db9dcdaba23/main/0.log" Jul 21 22:35:19.268044 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:19.268016 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-x4szz_5f80ef63-43c1-432f-8942-618092eb6709/main/0.log" Jul 21 22:35:24.824085 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.824039 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wffls/must-gather-9kmqb"] Jul 21 22:35:24.824669 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.824558 2557 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24c95235-0a91-4aa1-8b34-b49f53b1e1c6" containerName="test-curl-7616-139937563645968" Jul 21 22:35:24.824669 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.824577 2557 state_mem.go:107] "Deleted CPUSet assignment" podUID="24c95235-0a91-4aa1-8b34-b49f53b1e1c6" containerName="test-curl-7616-139937563645968" Jul 21 22:35:24.824800 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.824671 2557 memory_manager.go:368] "RemoveStaleState removing state" podUID="24c95235-0a91-4aa1-8b34-b49f53b1e1c6" containerName="test-curl-7616-139937563645968" Jul 21 22:35:24.828057 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.828033 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wffls/must-gather-9kmqb" Jul 21 22:35:24.831343 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.831321 2557 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-wffls\"/\"default-dockercfg-fk87s\"" Jul 21 22:35:24.832070 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.832052 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wffls\"/\"openshift-service-ca.crt\"" Jul 21 22:35:24.832174 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.832119 2557 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-wffls\"/\"kube-root-ca.crt\"" Jul 21 22:35:24.844863 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.844837 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wffls/must-gather-9kmqb"] Jul 21 22:35:24.863835 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.863810 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b38f58d9-95fe-44e6-a94f-27dd99fc66b7-must-gather-output\") pod \"must-gather-9kmqb\" (UID: \"b38f58d9-95fe-44e6-a94f-27dd99fc66b7\") " pod="openshift-must-gather-wffls/must-gather-9kmqb" Jul 21 22:35:24.863972 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.863848 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhfm8\" (UniqueName: \"kubernetes.io/projected/b38f58d9-95fe-44e6-a94f-27dd99fc66b7-kube-api-access-fhfm8\") pod \"must-gather-9kmqb\" (UID: \"b38f58d9-95fe-44e6-a94f-27dd99fc66b7\") " pod="openshift-must-gather-wffls/must-gather-9kmqb" Jul 21 22:35:24.964776 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.964741 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhfm8\" (UniqueName: \"kubernetes.io/projected/b38f58d9-95fe-44e6-a94f-27dd99fc66b7-kube-api-access-fhfm8\") pod \"must-gather-9kmqb\" (UID: \"b38f58d9-95fe-44e6-a94f-27dd99fc66b7\") " pod="openshift-must-gather-wffls/must-gather-9kmqb" Jul 21 22:35:24.964961 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.964829 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b38f58d9-95fe-44e6-a94f-27dd99fc66b7-must-gather-output\") pod \"must-gather-9kmqb\" (UID: \"b38f58d9-95fe-44e6-a94f-27dd99fc66b7\") " pod="openshift-must-gather-wffls/must-gather-9kmqb" Jul 21 22:35:24.965145 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.965129 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/b38f58d9-95fe-44e6-a94f-27dd99fc66b7-must-gather-output\") pod \"must-gather-9kmqb\" (UID: \"b38f58d9-95fe-44e6-a94f-27dd99fc66b7\") " pod="openshift-must-gather-wffls/must-gather-9kmqb" Jul 21 22:35:24.974693 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:24.974673 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhfm8\" (UniqueName: \"kubernetes.io/projected/b38f58d9-95fe-44e6-a94f-27dd99fc66b7-kube-api-access-fhfm8\") pod \"must-gather-9kmqb\" (UID: \"b38f58d9-95fe-44e6-a94f-27dd99fc66b7\") " pod="openshift-must-gather-wffls/must-gather-9kmqb" Jul 21 22:35:25.137136 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:25.137027 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wffls/must-gather-9kmqb" Jul 21 22:35:25.267957 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:25.267934 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wffls/must-gather-9kmqb"] Jul 21 22:35:25.270255 ip-10-0-138-36 kubenswrapper[2557]: W0721 22:35:25.270229 2557 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb38f58d9_95fe_44e6_a94f_27dd99fc66b7.slice/crio-99561879de78ba77c59e3ac3ad23e9de5beb50c4575d5e380e8524cdd0480f55 WatchSource:0}: Error finding container 99561879de78ba77c59e3ac3ad23e9de5beb50c4575d5e380e8524cdd0480f55: Status 404 returned error can't find the container with id 99561879de78ba77c59e3ac3ad23e9de5beb50c4575d5e380e8524cdd0480f55 Jul 21 22:35:25.647893 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:25.647858 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/must-gather-9kmqb" event={"ID":"b38f58d9-95fe-44e6-a94f-27dd99fc66b7","Type":"ContainerStarted","Data":"99561879de78ba77c59e3ac3ad23e9de5beb50c4575d5e380e8524cdd0480f55"} Jul 21 22:35:26.658249 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:26.657409 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/must-gather-9kmqb" event={"ID":"b38f58d9-95fe-44e6-a94f-27dd99fc66b7","Type":"ContainerStarted","Data":"b77e7831495d79230bb67cd81eaf2dacb8a5ae4f64d9e16cc9b9ad29ed40f817"} Jul 21 22:35:26.658249 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:26.657467 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/must-gather-9kmqb" event={"ID":"b38f58d9-95fe-44e6-a94f-27dd99fc66b7","Type":"ContainerStarted","Data":"45d34d4e847ccf02dd801de7be11fee4c346ec5d37e40ea840cb0156be26ae95"} Jul 21 22:35:26.679201 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:26.679148 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wffls/must-gather-9kmqb" podStartSLOduration=1.821993244 podStartE2EDuration="2.679131256s" totalImagesPullingTime="857.138012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 22:35:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:35:26.675870007 +0000 UTC m=+2826.781834154" watchObservedRunningTime="2026-07-21 22:35:26.679131256 +0000 UTC m=+2826.785095402" Jul 21 22:35:27.816850 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:27.816812 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-szw9m_c22209d7-ce32-4882-870d-1ec2f71fe97a/global-pull-secret-syncer/0.log" Jul 21 22:35:27.970405 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:27.970379 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-rbwsp_e43a69f9-df08-4cfc-9704-a153830b131e/konnectivity-agent/0.log" Jul 21 22:35:28.075654 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:28.075578 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-36.ec2.internal_e5af688db312735faccb6800239fbc06/haproxy/0.log" Jul 21 22:35:31.561540 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.561513 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k_10c555e8-5f68-4a6c-bae0-4223d26350d7/extract/0.log" Jul 21 22:35:31.593687 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.593662 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k_10c555e8-5f68-4a6c-bae0-4223d26350d7/util/0.log" Jul 21 22:35:31.622619 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.622568 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7592gm7k_10c555e8-5f68-4a6c-bae0-4223d26350d7/pull/0.log" Jul 21 22:35:31.686135 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.686109 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7_54dda3dd-b498-450e-880d-89c685642624/extract/0.log" Jul 21 22:35:31.710238 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.710205 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7_54dda3dd-b498-450e-880d-89c685642624/util/0.log" Jul 21 22:35:31.735656 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.735618 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0c5gx7_54dda3dd-b498-450e-880d-89c685642624/pull/0.log" Jul 21 22:35:31.767113 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.767038 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p_3da4e59f-0739-4ae2-a15d-8f08843a165e/extract/0.log" Jul 21 22:35:31.822220 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.822138 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p_3da4e59f-0739-4ae2-a15d-8f08843a165e/util/0.log" Jul 21 22:35:31.848152 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.848083 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed7327k5p_3da4e59f-0739-4ae2-a15d-8f08843a165e/pull/0.log" Jul 21 22:35:31.893959 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.893924 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245_5c647702-75d8-419b-9d6b-2b6ba74f5080/extract/0.log" Jul 21 22:35:31.938638 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.938614 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245_5c647702-75d8-419b-9d6b-2b6ba74f5080/util/0.log" Jul 21 22:35:31.968869 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:31.968847 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1jp245_5c647702-75d8-419b-9d6b-2b6ba74f5080/pull/0.log" Jul 21 22:35:32.446546 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:32.444554 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6fd888dcb8-zdcgs_defe2a6f-3e8b-4636-9549-1c4e1376c572/authorino/0.log" Jul 21 22:35:32.543838 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:32.543802 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-jr5pd_cf7bcd1e-4217-4651-8dfb-c33468138baa/kuadrant-console-plugin/0.log" Jul 21 22:35:32.742901 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:32.742865 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-lvrlh_fada2524-786d-4b4f-a0e9-81fe443006ad/limitador/0.log" Jul 21 22:35:34.828279 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:34.828247 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-9qff2_67567445-406c-4492-b0f3-1a484f765c4f/cluster-monitoring-operator/0.log" Jul 21 22:35:35.223878 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:35.223846 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5lft_61108f6f-64e9-4d13-936c-2e57d1607da0/node-exporter/0.log" Jul 21 22:35:35.248783 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:35.248750 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5lft_61108f6f-64e9-4d13-936c-2e57d1607da0/kube-rbac-proxy/0.log" Jul 21 22:35:35.274654 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:35.274620 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-s5lft_61108f6f-64e9-4d13-936c-2e57d1607da0/init-textfile/0.log" Jul 21 22:35:37.762553 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:37.762466 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/1.log" Jul 21 22:35:37.769392 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:37.769365 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-gh7jz_3bc3b474-56dc-422b-8cb5-26a5bc9ff5cc/console-operator/2.log" Jul 21 22:35:38.059261 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.059181 2557 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb"] Jul 21 22:35:38.066949 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.066916 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.073456 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.073425 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb"] Jul 21 22:35:38.218514 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.218480 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-podres\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.218709 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.218524 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-lib-modules\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.218709 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.218571 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-proc\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.218709 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.218700 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-sys\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.218896 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.218732 2557 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrtd6\" (UniqueName: \"kubernetes.io/projected/22f3711d-485f-4284-80a3-e0616ae1f777-kube-api-access-jrtd6\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.319920 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.319832 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-lib-modules\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.319920 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.319891 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-proc\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.320176 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.319958 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-sys\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.320176 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.319978 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-proc\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.320176 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.319982 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrtd6\" (UniqueName: \"kubernetes.io/projected/22f3711d-485f-4284-80a3-e0616ae1f777-kube-api-access-jrtd6\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.320176 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.320018 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-lib-modules\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.320176 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.320075 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-sys\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.320176 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.320113 2557 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-podres\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.320526 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.320227 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/22f3711d-485f-4284-80a3-e0616ae1f777-podres\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.329902 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.329858 2557 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrtd6\" (UniqueName: \"kubernetes.io/projected/22f3711d-485f-4284-80a3-e0616ae1f777-kube-api-access-jrtd6\") pod \"perf-node-gather-daemonset-2kfgb\" (UID: \"22f3711d-485f-4284-80a3-e0616ae1f777\") " pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.372244 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.372210 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-q78cf_3d5455ad-df25-46af-9e32-c725ed4267c7/download-server/0.log" Jul 21 22:35:38.382434 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.382409 2557 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.577163 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.576942 2557 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb"] Jul 21 22:35:38.758783 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.758741 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" event={"ID":"22f3711d-485f-4284-80a3-e0616ae1f777","Type":"ContainerStarted","Data":"cb0ec5d5064ab9e5b2a7725523c7c70a12610f828b3ed6f068920d5738b15fd9"} Jul 21 22:35:38.759072 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.759054 2557 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" event={"ID":"22f3711d-485f-4284-80a3-e0616ae1f777","Type":"ContainerStarted","Data":"718e81cc0e49a287b6b88b52655cdb055aeb77e0a4e76c0dfbb5dad5e0eae758"} Jul 21 22:35:38.759265 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.759251 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:38.779876 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:38.779818 2557 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" podStartSLOduration=0.779797248 podStartE2EDuration="779.797248ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 22:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 22:35:38.77681505 +0000 UTC m=+2838.882779217" watchObservedRunningTime="2026-07-21 22:35:38.779797248 +0000 UTC m=+2838.885761396" Jul 21 22:35:39.064224 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:39.064190 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-j7cc4_9a0ff853-f1f9-44b2-96df-6f97300e61ab/volume-data-source-validator/0.log" Jul 21 22:35:40.185463 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:40.185432 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ck8nd_36237ef1-41a3-4fe3-bf11-b2f5712df413/dns/0.log" Jul 21 22:35:40.214756 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:40.214728 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-ck8nd_36237ef1-41a3-4fe3-bf11-b2f5712df413/kube-rbac-proxy/0.log" Jul 21 22:35:40.344216 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:40.344183 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gptzn_03fb8246-7a76-492f-9e8b-b658d2a5fd49/dns-node-resolver/0.log" Jul 21 22:35:40.978729 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:40.978701 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-cdjcd_418b4e01-6bac-4840-8164-88a4f0ef298b/node-ca/0.log" Jul 21 22:35:42.304365 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:42.304333 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594787b58-x8sdj_50b67417-ddfd-4dd5-bede-d132bc7752f2/kube-auth-proxy/0.log" Jul 21 22:35:42.461228 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:42.461192 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8649769886-nbx92_37e21193-957c-4af6-b708-b77445d6229a/router/0.log" Jul 21 22:35:43.184213 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:43.184184 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2cczj_6916be75-1fca-4fbd-8bf1-d210b4416423/serve-healthcheck-canary/0.log" Jul 21 22:35:43.747980 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:43.747946 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-2hqv9_25671a47-f968-4834-8534-1f0e3e92f43b/insights-operator/0.log" Jul 21 22:35:43.749443 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:43.749423 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-2hqv9_25671a47-f968-4834-8534-1f0e3e92f43b/insights-operator/1.log" Jul 21 22:35:43.921493 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:43.921465 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbgfp_3d1a845f-de5b-4748-866c-87b10b7ee243/kube-rbac-proxy/0.log" Jul 21 22:35:43.946887 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:43.946861 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbgfp_3d1a845f-de5b-4748-866c-87b10b7ee243/exporter/0.log" Jul 21 22:35:43.974784 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:43.974752 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-zbgfp_3d1a845f-de5b-4748-866c-87b10b7ee243/extractor/0.log" Jul 21 22:35:44.778740 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:44.778710 2557 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-wffls/perf-node-gather-daemonset-2kfgb" Jul 21 22:35:46.576223 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:46.576194 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zr6gw_14f720e4-99c1-4618-8be1-28a4f2881372/server/0.log" Jul 21 22:35:46.672023 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:46.671990 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-n4qw9_47ef4f44-ca7b-4104-9041-143b18022eb1/manager/0.log" Jul 21 22:35:46.768643 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:46.768603 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-5sfcn_63c8eb32-068c-420d-82ca-2a4cbe3cb4ab/manager/0.log" Jul 21 22:35:46.788112 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:46.788065 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54667d85dc-5sfcn_63c8eb32-068c-420d-82ca-2a4cbe3cb4ab/copy-manifests/0.log" Jul 21 22:35:48.393696 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:48.393615 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-d89c66ff-dvzhk_17f04541-80bc-48d5-9d51-7e7315092316/manager/0.log" Jul 21 22:35:48.651892 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:48.651811 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594787b58-x8sdj_50b67417-ddfd-4dd5-bede-d132bc7752f2/kube-auth-proxy/0.log" Jul 21 22:35:48.753148 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:48.753118 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8649769886-nbx92_37e21193-957c-4af6-b708-b77445d6229a/router/0.log" Jul 21 22:35:50.247898 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:50.247855 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-594787b58-x8sdj_50b67417-ddfd-4dd5-bede-d132bc7752f2/kube-auth-proxy/0.log" Jul 21 22:35:50.376574 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:50.376503 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-8649769886-nbx92_37e21193-957c-4af6-b708-b77445d6229a/router/0.log" Jul 21 22:35:54.163191 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:54.163152 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-smvjt_794afb78-cd2f-4061-8a6b-950b5c00240f/kube-storage-version-migrator-operator/1.log" Jul 21 22:35:54.164132 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:54.164113 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-7658d9b749-smvjt_794afb78-cd2f-4061-8a6b-950b5c00240f/kube-storage-version-migrator-operator/0.log" Jul 21 22:35:55.527562 ip-10-0-138-36 kubenswrapper[2557]: I0721 22:35:55.527532 2557 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9xcsz_8b8b5eb2-9602-4550-9841-d06829964c04/kube-multus-additional-cni-plugins/0.log"