Jul 08 16:03:16.150528 ip-10-0-141-121 systemd[1]: Starting Kubernetes Kubelet... Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.624554 2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631727 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631744 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631747 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631751 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631754 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631756 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631759 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 16:03:16.641564 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631761 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631764 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631767 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631771 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631774 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631776 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631779 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631782 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631784 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631787 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631790 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631793 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631802 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631805 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631807 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631810 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631812 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631815 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631817 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631820 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 16:03:16.651596 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631824 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631826 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631829 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631832 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631834 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631837 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631839 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631842 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631844 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631847 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631849 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631852 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631854 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631857 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631860 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631864 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631869 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631873 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631877 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 16:03:16.654677 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631880 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631883 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631885 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631888 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631891 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631895 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631898 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631900 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631903 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631905 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631908 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631911 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631913 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631916 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631919 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631921 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631924 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631926 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631929 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631932 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 16:03:16.666477 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631934 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631937 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631940 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631942 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631946 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631951 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631954 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631958 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631961 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631963 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631966 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631972 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631976 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631980 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631983 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631986 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631991 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631995 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.631998 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 16:03:16.670393 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632002 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632005 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632009 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632013 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632018 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632022 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632025 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632030 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632034 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632041 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632045 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632049 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632053 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632057 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632959 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632966 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632969 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632971 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632974 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632977 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 16:03:16.687174 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632980 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632982 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632985 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632988 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632991 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.632999 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633002 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633005 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633008 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633013 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633017 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633022 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633027 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633030 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633034 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633038 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633041 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633043 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633046 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633048 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 16:03:16.688944 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633051 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633053 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633056 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633059 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633061 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633064 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633066 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633069 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633071 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633074 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633076 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633079 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633081 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633084 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633087 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633090 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633092 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633095 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633103 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633106 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 16:03:16.708307 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633109 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633113 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633131 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633136 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633139 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633141 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633144 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633148 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633152 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633155 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633158 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633161 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633163 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633166 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633169 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633171 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633174 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633176 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633179 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 16:03:16.711848 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633182 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633184 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633187 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633189 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633192 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633194 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633197 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633201 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633203 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633206 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633208 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633211 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633215 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633217 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633221 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633224 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633227 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633229 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633232 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 16:03:16.728520 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633234 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 16:03:16.713986 ip-10-0-141-121 systemd[1]: Started Kubernetes Kubelet. Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633238 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633241 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633244 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633246 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633249 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633252 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633255 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633259 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633261 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633264 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633267 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633269 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633272 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633275 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633664 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633668 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633671 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633674 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633678 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 16:03:16.732770 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633680 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633683 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633686 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633688 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633691 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633694 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633697 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633701 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633704 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633706 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633709 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633713 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633716 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633718 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633721 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633724 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633726 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633729 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633731 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633734 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 16:03:16.745940 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633737 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633739 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633741 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633744 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633747 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633749 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633752 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633755 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633757 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633761 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633765 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633768 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633771 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633773 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633776 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633778 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633781 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633784 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633787 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633791 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 16:03:16.896866 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633794 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633796 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633799 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633803 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633806 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633808 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633811 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633814 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633817 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633819 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633822 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633824 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633827 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633829 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633832 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633835 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633839 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633843 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633846 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633849 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 16:03:16.916994 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633852 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633854 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633857 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633860 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633862 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633865 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633867 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633870 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633873 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633875 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633878 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633882 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633885 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633887 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633891 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633894 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633897 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633900 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633903 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633905 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 16:03:16.919699 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633908 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633910 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633913 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633915 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633918 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633920 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633923 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633925 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633928 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633930 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633933 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633937 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633941 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.633945 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634016 2563 flags.go:64] FLAG: --address="0.0.0.0" Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634024 2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634031 2563 flags.go:64] FLAG: --anonymous-auth="true" Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634036 2563 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634040 2563 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634043 2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 08 16:03:16.928569 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634047 2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634052 2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634055 2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634059 2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634063 2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634067 2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634070 2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634075 2563 flags.go:64] FLAG: --cgroup-root="" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634078 2563 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634081 2563 flags.go:64] FLAG: --client-ca-file="" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634084 2563 flags.go:64] FLAG: --cloud-provider="external" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634087 2563 flags.go:64] FLAG: --cluster-dns="[]" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634091 2563 flags.go:64] FLAG: --cluster-domain="" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634094 2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634097 2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634100 2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634104 2563 flags.go:64] FLAG: --container-log-max-files="5" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634107 2563 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634110 2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634114 2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634135 2563 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634140 2563 flags.go:64] FLAG: --contention-profiling="false" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634143 2563 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634147 2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 08 16:03:16.929631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634151 2563 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634155 2563 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634160 2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634163 2563 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634167 2563 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634170 2563 flags.go:64] FLAG: --enable-load-reader="false" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634174 2563 flags.go:64] FLAG: --enable-server="true" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634178 2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634184 2563 flags.go:64] FLAG: --event-burst="100" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634187 2563 flags.go:64] FLAG: --event-qps="50" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634191 2563 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634196 2563 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634200 2563 flags.go:64] FLAG: --eviction-hard="" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634205 2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634209 2563 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634214 2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634218 2563 flags.go:64] FLAG: --eviction-soft="" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634222 2563 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634225 2563 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634229 2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634233 2563 flags.go:64] FLAG: --experimental-mounter-path="" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634236 2563 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634240 2563 flags.go:64] FLAG: --fail-swap-on="true" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634244 2563 flags.go:64] FLAG: --feature-gates="" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634248 2563 flags.go:64] FLAG: --file-check-frequency="20s" Jul 08 16:03:16.930423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634252 2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634256 2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634260 2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634264 2563 flags.go:64] FLAG: --healthz-port="10248" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634267 2563 flags.go:64] FLAG: --help="false" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634271 2563 flags.go:64] FLAG: --hostname-override="ip-10-0-141-121.ec2.internal" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634275 2563 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634279 2563 flags.go:64] FLAG: --http-check-frequency="20s" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634283 2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634287 2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634291 2563 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634295 2563 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634298 2563 flags.go:64] FLAG: --image-service-endpoint="" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634302 2563 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634305 2563 flags.go:64] FLAG: --kube-api-burst="100" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634308 2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634312 2563 flags.go:64] FLAG: --kube-api-qps="50" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634316 2563 flags.go:64] FLAG: --kube-reserved="" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634321 2563 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634324 2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634328 2563 flags.go:64] FLAG: --kubelet-cgroups="" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634332 2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634338 2563 flags.go:64] FLAG: --lock-file="" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634341 2563 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 08 16:03:16.932705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634345 2563 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634349 2563 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634356 2563 flags.go:64] FLAG: --log-json-split-stream="false" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634359 2563 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634363 2563 flags.go:64] FLAG: --log-text-split-stream="false" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634366 2563 flags.go:64] FLAG: --logging-format="text" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634370 2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634374 2563 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634377 2563 flags.go:64] FLAG: --manifest-url="" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634381 2563 flags.go:64] FLAG: --manifest-url-header="" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634386 2563 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634390 2563 flags.go:64] FLAG: --max-open-files="1000000" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634395 2563 flags.go:64] FLAG: --max-pods="110" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634399 2563 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634403 2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634406 2563 flags.go:64] FLAG: --memory-manager-policy="None" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634409 2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634413 2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634417 2563 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634420 2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634429 2563 flags.go:64] FLAG: --node-status-max-images="50" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634433 2563 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634437 2563 flags.go:64] FLAG: --oom-score-adj="-999" Jul 08 16:03:16.933494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634440 2563 flags.go:64] FLAG: --pod-cidr="" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634444 2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634452 2563 flags.go:64] FLAG: --pod-manifest-path="" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634458 2563 flags.go:64] FLAG: --pod-max-pids="-1" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634462 2563 flags.go:64] FLAG: --pods-per-core="0" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634467 2563 flags.go:64] FLAG: --port="10250" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634472 2563 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634477 2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a9816d1a4324eb76" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634484 2563 flags.go:64] FLAG: --qos-reserved="" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634488 2563 flags.go:64] FLAG: --read-only-port="10255" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634492 2563 flags.go:64] FLAG: --register-node="true" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634497 2563 flags.go:64] FLAG: --register-with-taints="" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634503 2563 flags.go:64] FLAG: --registry-burst="10" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634507 2563 flags.go:64] FLAG: --registry-qps="5" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634512 2563 flags.go:64] FLAG: --reserved-cpus="" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634517 2563 flags.go:64] FLAG: --reserved-memory="" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634523 2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634528 2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634533 2563 flags.go:64] FLAG: --rotate-certificates="false" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634538 2563 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634542 2563 flags.go:64] FLAG: --runonce="false" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634547 2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634552 2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634557 2563 flags.go:64] FLAG: --seccomp-default="false" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634561 2563 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 08 16:03:16.934299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634565 2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634570 2563 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634575 2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634581 2563 flags.go:64] FLAG: --storage-driver-password="root" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634586 2563 flags.go:64] FLAG: --storage-driver-secure="false" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634590 2563 flags.go:64] FLAG: --storage-driver-table="stats" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634594 2563 flags.go:64] FLAG: --storage-driver-user="root" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634599 2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634603 2563 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634608 2563 flags.go:64] FLAG: --system-cgroups="" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634614 2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634623 2563 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634627 2563 flags.go:64] FLAG: --tls-cert-file="" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634631 2563 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634637 2563 flags.go:64] FLAG: --tls-min-version="" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634644 2563 flags.go:64] FLAG: --tls-private-key-file="" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634648 2563 flags.go:64] FLAG: --topology-manager-policy="none" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634652 2563 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634657 2563 flags.go:64] FLAG: --topology-manager-scope="container" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634661 2563 flags.go:64] FLAG: --v="2" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634674 2563 flags.go:64] FLAG: --version="false" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634680 2563 flags.go:64] FLAG: --vmodule="" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634686 2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.634691 2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634847 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 16:03:16.935231 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634852 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634855 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634858 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634862 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634864 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634867 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634870 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634872 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634875 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634878 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634880 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634883 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634886 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634888 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634891 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634894 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634897 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634901 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634905 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634909 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 16:03:16.936006 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634912 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634915 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634919 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634922 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634925 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634927 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634930 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634933 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634935 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634938 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634941 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634943 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634946 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634948 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634951 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634954 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634956 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634959 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634962 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634964 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 16:03:16.937077 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634967 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634969 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634972 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634974 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634977 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634979 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634982 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634984 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634987 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634990 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634993 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634996 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.634998 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635001 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635005 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635007 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635012 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635014 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635017 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635020 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 16:03:16.937656 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635022 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635025 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635027 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635030 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635033 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635035 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635038 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635041 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635043 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635046 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635048 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635051 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635055 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635058 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635061 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635064 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635066 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635069 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635072 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 16:03:17.138248 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635074 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635077 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635080 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635083 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635085 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635088 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635092 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635096 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635099 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635103 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635106 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635109 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635112 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635114 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635134 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635137 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635139 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635142 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 16:03:17.138867 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.635146 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.635985 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.645066 2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.645082 2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.645096 2563 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.645103 2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645180 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645187 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645190 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645194 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645197 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645200 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645203 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645207 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645210 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 16:03:17.410901 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645212 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645215 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645218 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645222 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645224 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645228 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645230 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645233 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645236 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645238 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645242 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645245 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645248 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645250 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645254 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645256 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645259 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645263 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645265 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645268 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 16:03:17.412110 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645271 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645274 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645277 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645279 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645282 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645284 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645287 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645290 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645293 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645295 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645298 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645301 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645303 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645306 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645309 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645311 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645314 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645317 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645319 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645322 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 16:03:17.412690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645325 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645330 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645334 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645337 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645341 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645344 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645348 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645351 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645354 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645357 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645359 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645362 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645365 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645368 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645370 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645373 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645376 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645378 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645381 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 16:03:17.413330 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645384 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645387 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645389 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645392 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645395 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645397 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645400 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645403 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645405 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645408 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645411 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645414 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645417 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645419 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645422 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645425 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645428 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645431 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645433 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 16:03:17.413838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645436 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645438 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645441 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645444 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645446 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645449 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645452 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645455 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645459 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645466 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645470 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645473 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.645478 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645597 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 08 16:03:17.414415 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645603 2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645606 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645609 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645612 2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645616 2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645620 2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645623 2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645626 2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645629 2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645632 2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645634 2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645638 2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645641 2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645644 2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645647 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645651 2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645654 2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645658 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645660 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 08 16:03:17.414838 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645663 2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645667 2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645671 2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645674 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645677 2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645680 2563 feature_gate.go:328] unrecognized feature gate: Example Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645682 2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645685 2563 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645688 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645692 2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645695 2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645698 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645701 2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645703 2563 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645706 2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645708 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645711 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645713 2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645716 2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645719 2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 08 16:03:17.415423 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645721 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645724 2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645727 2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645729 2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645732 2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645735 2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645738 2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645741 2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645744 2563 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645747 2563 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645749 2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645752 2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645754 2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645757 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645760 2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645763 2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645765 2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645768 2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645771 2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645773 2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 08 16:03:17.416100 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645776 2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645780 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645783 2563 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645786 2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645788 2563 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645791 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645793 2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645796 2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645799 2563 feature_gate.go:328] unrecognized feature gate: Example2 Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645801 2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645804 2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645807 2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645809 2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645812 2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645815 2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645817 2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645820 2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645822 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645826 2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645829 2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 08 16:03:17.416647 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645832 2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645834 2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645837 2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645839 2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645842 2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645845 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645847 2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645850 2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645853 2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645855 2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645858 2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645861 2563 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645863 2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645869 2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645872 2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645875 2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645878 2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645880 2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 08 16:03:17.417142 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:16.645883 2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.645888 2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.646001 2563 server.go:962] "Client rotation is on, will bootstrap in background" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.648468 2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.650041 2563 server.go:1019] "Starting client certificate rotation" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.650152 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.650186 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.652739 2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.655052 2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.671135 2563 log.go:25] "Validated CRI v1 runtime API" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.673924 2563 log.go:25] "Validated CRI v1 image API" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.674897 2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 08 16:03:17.417585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.680083 2563 fs.go:135] Filesystem UUIDs: map[0b8590f5-607a-4708-a6ae-ce53b9e99cdc:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 99c86249-3aac-4399-88b6-2621559aefbd:/dev/nvme0n1p3] Jul 08 16:03:17.417888 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.680100 2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 08 16:03:17.417888 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.681315 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.687561 2563 manager.go:216] Machine: {Timestamp:2026-07-08 16:03:16.685962534 +0000 UTC m=+0.393752373 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3089656 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20a0f95479d8418b63a70e7fadebe3 SystemUUID:ec20a0f9-5479-d841-8b63-a70e7fadebe3 BootID:6f218622-321b-4975-9879-7cb68c780b2b Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:0a:62:c8:c9:e3 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:0a:62:c8:c9:e3 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:3a:58:0e:3e:b2:b1 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.687662 2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.687748 2563 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.691572 2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.691606 2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-121.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.692728 2563 topology_manager.go:138] "Creating topology manager with none policy" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.692742 2563 container_manager_linux.go:309] "Creating device plugin manager" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.692750 2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.692768 2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.694550 2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.698516 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.698674 2563 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.698707 2563 kubelet.go:486] "Attempting to sync node with API server" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.698724 2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.698742 2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.698755 2563 kubelet.go:398] "Adding apiserver pod source" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.698764 2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 08 16:03:17.417953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.699280 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.699307 2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.703007 2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.705773 2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.705804 2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.707890 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.707915 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.707930 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.707938 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.707955 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.707964 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.707980 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.707994 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.708004 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.708013 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.708029 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.708043 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.709895 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.709909 2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.710633 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.710633 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.713064 2563 server.go:1268] "Started kubelet" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.713147 2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.713575 2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.713641 2563 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.715249 2563 server.go:310] "Adding debug handlers to kubelet server" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.715570 2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.719457 2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.723144 2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.723161 2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.723722 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.724030 2563 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.724042 2563 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.724110 2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.724192 2563 reconstruct.go:97] "Volume reconstruction finished" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.724199 2563 reconciler.go:29] "Reconciler: start to sync state" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.725065 2563 factory.go:55] Registering systemd factory Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.725082 2563 factory.go:223] Registration of the systemd container factory successfully Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.726044 2563 factory.go:153] Registering CRI-O factory Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.726058 2563 factory.go:223] Registration of the crio container factory successfully Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.726140 2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.726163 2563 factory.go:103] Registering Raw factory Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.726177 2563 manager.go:1196] Started watching for new ooms in manager Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.726548 2563 manager.go:349] Starting recovery of all containers Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.726705 2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 08 16:03:17.418449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.726771 2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-121.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.726846 2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-121.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.726165 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-121.ec2.internal.18c05bfd7c8ab2a8 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-121.ec2.internal,UID:ip-10-0-141-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-121.ec2.internal,},FirstTimestamp:2026-07-08 16:03:16.713034408 +0000 UTC m=+0.420824245,LastTimestamp:2026-07-08 16:03:16.713034408 +0000 UTC m=+0.420824245,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-121.ec2.internal,}" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.740691 2563 manager.go:354] Recovery completed Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.741904 2563 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.744912 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.746905 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientMemory" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.746948 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.746963 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientPID" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.747549 2563 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.747560 2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.747564 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.747575 2563 state_mem.go:36] "Initialized new in-memory state store" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.748297 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientMemory" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.748326 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.748341 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientPID" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.748764 2563 policy_none.go:49] "None policy: Start" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.748780 2563 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.748797 2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.749826 2563 policy_none.go:47] "Start" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.753831 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xmkz4" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.758398 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-121.ec2.internal.18c05bfd7e8fd69c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-121.ec2.internal,UID:ip-10-0-141-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-121.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-121.ec2.internal,},FirstTimestamp:2026-07-08 16:03:16.746925724 +0000 UTC m=+0.454715565,LastTimestamp:2026-07-08 16:03:16.746925724 +0000 UTC m=+0.454715565,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-121.ec2.internal,}" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.766269 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-xmkz4" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.768522 2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-121.ec2.internal.18c05bfd7e904998 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-121.ec2.internal,UID:ip-10-0-141-121.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-121.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-121.ec2.internal,},FirstTimestamp:2026-07-08 16:03:16.74695516 +0000 UTC m=+0.454744999,LastTimestamp:2026-07-08 16:03:16.74695516 +0000 UTC m=+0.454744999,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-121.ec2.internal,}" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.774748 2563 manager.go:337] "Starting Device Plugin manager" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.774780 2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 08 16:03:17.419578 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.774789 2563 server.go:87] "Starting device plugin registration server" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.775009 2563 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.775019 2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.775089 2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.775176 2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.775184 2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.775650 2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.775700 2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.833561 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.834721 2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.834759 2563 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.834781 2563 kubelet.go:2448] "Starting kubelet main sync loop" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.834816 2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.837201 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.875458 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.876340 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientMemory" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.876368 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.876379 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientPID" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.876400 2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.886233 2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.886256 2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-121.ec2.internal\": node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.932814 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.935115 2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal"] Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.935739 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.936996 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientMemory" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.937024 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.937034 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientPID" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.938242 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.938367 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.938397 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.938900 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientMemory" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.938914 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientMemory" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.938934 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.938943 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientPID" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.938920 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.939028 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientPID" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.940796 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.940816 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.941451 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientMemory" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.941466 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:16.941475 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientPID" Jul 08 16:03:17.420256 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.972705 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-121.ec2.internal\" not found" node="ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:16.977366 2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-121.ec2.internal\" not found" node="ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.025385 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f73b543c0aedc66b9e736106d082533-config\") pod \"kube-apiserver-proxy-ip-10-0-141-121.ec2.internal\" (UID: \"0f73b543c0aedc66b9e736106d082533\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.025407 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/907e537dbcd976156b6c67c78c05344d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal\" (UID: \"907e537dbcd976156b6c67c78c05344d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.025426 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/907e537dbcd976156b6c67c78c05344d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal\" (UID: \"907e537dbcd976156b6c67c78c05344d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:17.032975 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.125781 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/907e537dbcd976156b6c67c78c05344d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal\" (UID: \"907e537dbcd976156b6c67c78c05344d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.125838 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/907e537dbcd976156b6c67c78c05344d-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal\" (UID: \"907e537dbcd976156b6c67c78c05344d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.125861 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f73b543c0aedc66b9e736106d082533-config\") pod \"kube-apiserver-proxy-ip-10-0-141-121.ec2.internal\" (UID: \"0f73b543c0aedc66b9e736106d082533\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.125885 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/907e537dbcd976156b6c67c78c05344d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal\" (UID: \"907e537dbcd976156b6c67c78c05344d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.125919 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/907e537dbcd976156b6c67c78c05344d-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal\" (UID: \"907e537dbcd976156b6c67c78c05344d\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.125943 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/0f73b543c0aedc66b9e736106d082533-config\") pod \"kube-apiserver-proxy-ip-10-0-141-121.ec2.internal\" (UID: \"0f73b543c0aedc66b9e736106d082533\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:17.133888 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:17.234638 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.275791 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:17.280656 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" Jul 08 16:03:17.421252 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:17.335064 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:17.435577 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:17.435553 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:18.699764 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:18.696947 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:18.797445 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:18.797408 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:18.811570 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:18.811552 2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 08 16:03:18.811994 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:18.811698 2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 08 16:03:18.897735 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:18.897667 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:18.928754 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:18.928707 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-07 15:58:17 +0000 UTC" deadline="2028-01-04 07:00:20.849083515 +0000 UTC" Jul 08 16:03:18.928754 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:18.928745 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13070h57m1.920342276s" Jul 08 16:03:18.964028 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:18.964005 2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 08 16:03:18.965119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:18.965106 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientMemory" Jul 08 16:03:18.965178 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:18.965134 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasNoDiskPressure" Jul 08 16:03:18.965178 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:18.965146 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeHasSufficientPID" Jul 08 16:03:18.997800 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:18.997784 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:19.098453 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.098423 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:19.190340 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.190266 2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 16:03:19.198711 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.198686 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:19.299152 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.299119 2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-121.ec2.internal\" not found" Jul 08 16:03:19.320348 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.320322 2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 16:03:19.334173 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.334154 2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 16:03:19.359808 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:19.359781 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod907e537dbcd976156b6c67c78c05344d.slice/crio-a9b6bfb5560792e4021e9b810a5bf303b6d46098cb67cc6486dbf22939e137d1 WatchSource:0}: Error finding container a9b6bfb5560792e4021e9b810a5bf303b6d46098cb67cc6486dbf22939e137d1: Status 404 returned error can't find the container with id a9b6bfb5560792e4021e9b810a5bf303b6d46098cb67cc6486dbf22939e137d1 Jul 08 16:03:19.360398 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:19.360369 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f73b543c0aedc66b9e736106d082533.slice/crio-fb33398fb504c3620214bbcff554db16fbc29fa0a04eee153c80f3064e2bdd53 WatchSource:0}: Error finding container fb33398fb504c3620214bbcff554db16fbc29fa0a04eee153c80f3064e2bdd53: Status 404 returned error can't find the container with id fb33398fb504c3620214bbcff554db16fbc29fa0a04eee153c80f3064e2bdd53 Jul 08 16:03:19.365372 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.365358 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 16:03:19.385258 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.385238 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" Jul 08 16:03:19.400145 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.400122 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 16:03:19.401110 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.401096 2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" Jul 08 16:03:19.412792 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.412771 2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 08 16:03:19.861231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.861198 2563 apiserver.go:52] "Watching apiserver" Jul 08 16:03:19.871664 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.871641 2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 08 16:03:19.872822 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.872798 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-x9p9z","openshift-ovn-kubernetes/ovnkube-node-g74wh","kube-system/global-pull-secret-syncer-wvsd9","openshift-cluster-node-tuning-operator/tuned-f5tfl","openshift-dns/node-resolver-jnp8q","openshift-image-registry/node-ca-27pnz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal","openshift-multus/multus-additional-cni-plugins-ztg8s","openshift-multus/multus-pxnkf","openshift-network-diagnostics/network-check-target-gfhcb","kube-system/konnectivity-agent-zkbjr","kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8","openshift-multus/network-metrics-daemon-8s9kq"] Jul 08 16:03:19.874147 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.874133 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:19.874214 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.874195 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:19.875279 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.875260 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.876421 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.876400 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:19.876514 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.876496 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:19.878143 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.878118 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.878519 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.878501 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 08 16:03:19.878519 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.878514 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 08 16:03:19.879225 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.878592 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-x462d\"" Jul 08 16:03:19.879225 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.878570 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 08 16:03:19.880510 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.880487 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:19.880701 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.880684 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 08 16:03:19.881396 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.881377 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 08 16:03:19.881549 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.881532 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tqcc9\"" Jul 08 16:03:19.882037 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.881778 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:19.883025 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.883005 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 08 16:03:19.883166 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.883145 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.883232 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.883183 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z7xwj\"" Jul 08 16:03:19.883497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.883384 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 08 16:03:19.884777 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.884757 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:19.884868 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.884837 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:19.885229 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.885172 2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 08 16:03:19.885764 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.885744 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 08 16:03:19.886406 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.885796 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 08 16:03:19.886520 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.886487 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 08 16:03:19.886596 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.885894 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 08 16:03:19.886659 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.885947 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-b9wbw\"" Jul 08 16:03:19.886711 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.886210 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 08 16:03:19.886781 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.886294 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 08 16:03:19.886974 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.886953 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.887056 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.887020 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-5hg56\"" Jul 08 16:03:19.887208 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.887192 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 08 16:03:19.887970 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.887952 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:19.889465 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.889449 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:19.890905 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.890820 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:19.891193 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.891171 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 08 16:03:19.891272 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.891207 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 08 16:03:19.891272 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.891213 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 08 16:03:19.891272 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.891224 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 08 16:03:19.891272 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.891181 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 08 16:03:19.892227 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.892211 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 08 16:03:19.892425 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.892328 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 08 16:03:19.893076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.892537 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4ftsg\"" Jul 08 16:03:19.893076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.892552 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 08 16:03:19.893076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.892632 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 08 16:03:19.893076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.892837 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tqf7x\"" Jul 08 16:03:19.893076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.892931 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 08 16:03:19.893076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.892950 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 08 16:03:19.893076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893003 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-rdr6c\"" Jul 08 16:03:19.893076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893054 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-cni-netd\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893128 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893172 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-socket-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-conf-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893233 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-etc-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893257 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-lib-modules\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893283 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jxdj\" (UniqueName: \"kubernetes.io/projected/849b9b0e-e00b-4793-aba1-38110bc88117-kube-api-access-4jxdj\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893308 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-socket-dir-parent\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893350 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-multus-certs\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893379 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-etc-kubernetes\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893405 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-var-lib-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893429 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-os-release\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893451 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-slash\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-modprobe-d\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.893497 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893495 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/849b9b0e-e00b-4793-aba1-38110bc88117-serviceca\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893518 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-cni-binary-copy\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893540 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-netns\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893563 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-cni-bin\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893620 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-systemd-units\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893651 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysconfig\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893674 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-kubernetes\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893698 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-kubelet\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893724 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-td7wb\" (UniqueName: \"kubernetes.io/projected/8a6cf69c-8067-47bf-b481-a1691e96d9ad-kube-api-access-td7wb\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893748 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-env-overrides\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893774 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zg7\" (UniqueName: \"kubernetes.io/projected/d05883b4-7ca2-49d8-80da-a2ba77d81548-kube-api-access-p7zg7\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893807 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-ovn\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893893 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovn-node-metrics-cert\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893937 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysctl-d\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.893965 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9dn\" (UniqueName: \"kubernetes.io/projected/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-kube-api-access-ft9dn\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894003 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-sys-fs\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-run\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894050 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c48df59e-0f52-49f7-8803-7038efe0952f-etc-tuned\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894117 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-cnibin\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894180 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovnkube-script-lib\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894204 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-host\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894226 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-hostroot\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894258 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6057259-93e1-4083-b7c0-863ea9fff32c-tmp-dir\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894294 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-log-socket\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894319 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-cni-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894354 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-kubelet\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894403 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-kubelet-config\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894430 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-dbus\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894458 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-sys\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894482 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-k8s-cni-cncf-io\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894505 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-systemd\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894527 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovnkube-config\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.894802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894556 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-phcks\" (UniqueName: \"kubernetes.io/projected/d5663b42-29a9-4cc9-9927-f401c43c45d7-kube-api-access-phcks\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894578 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894596 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-registration-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894610 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894626 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-device-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894640 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysctl-conf\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894654 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c48df59e-0f52-49f7-8803-7038efe0952f-tmp\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894678 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhkc7\" (UniqueName: \"kubernetes.io/projected/c48df59e-0f52-49f7-8803-7038efe0952f-kube-api-access-jhkc7\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894718 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-cni-multus\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894737 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-node-log\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894759 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/849b9b0e-e00b-4793-aba1-38110bc88117-host\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894781 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-dk68r\"" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894781 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6057259-93e1-4083-b7c0-863ea9fff32c-hosts-file\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894968 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-system-cni-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.894993 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895016 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-run-netns\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895047 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-systemd\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.895603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895085 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-var-lib-kubelet\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.896355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895108 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-daemon-config\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.896355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895131 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-cni-bin\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.896355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895141 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 08 16:03:19.896355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895154 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-etc-selinux\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.896355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895177 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:19.896355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895215 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg57v\" (UniqueName: \"kubernetes.io/projected/e6057259-93e1-4083-b7c0-863ea9fff32c-kube-api-access-mg57v\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:19.896355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.895424 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 08 16:03:19.901765 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.901745 2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 08 16:03:19.931297 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.931191 2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-qcttq" Jul 08 16:03:19.940564 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.940538 2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-qcttq" Jul 08 16:03:19.986600 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.986573 2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 08 16:03:19.995758 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995726 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:19.995931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995770 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-registration-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.995931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995800 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.995931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995824 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-device-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.995931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995847 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysctl-conf\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.995931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995872 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c48df59e-0f52-49f7-8803-7038efe0952f-tmp\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.995931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995894 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhkc7\" (UniqueName: \"kubernetes.io/projected/c48df59e-0f52-49f7-8803-7038efe0952f-kube-api-access-jhkc7\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.995931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-cni-multus\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995944 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-cni-binary-copy\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995973 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.995999 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-node-log\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996020 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/849b9b0e-e00b-4793-aba1-38110bc88117-host\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996043 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6057259-93e1-4083-b7c0-863ea9fff32c-hosts-file\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996087 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-system-cni-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-run-netns\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996166 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-systemd\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996189 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-var-lib-kubelet\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.996238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996218 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-daemon-config\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996243 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbq62\" (UniqueName: \"kubernetes.io/projected/32b8c356-9ba0-4e24-81e0-aab17229aab6-kube-api-access-dbq62\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996279 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-cni-bin\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996285 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-registration-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996305 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-etc-selinux\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996329 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996338 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996357 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg57v\" (UniqueName: \"kubernetes.io/projected/e6057259-93e1-4083-b7c0-863ea9fff32c-kube-api-access-mg57v\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996369 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-device-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996382 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-os-release\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996407 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996432 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-cni-netd\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996483 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysctl-conf\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996492 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996516 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-socket-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.996575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996540 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-conf-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996572 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32b8c356-9ba0-4e24-81e0-aab17229aab6-iptables-alerter-script\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996611 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-system-cni-dir\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996649 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996708 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-cni-bin\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996818 2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996826 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-etc-selinux\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.996922 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996945 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-cni-netd\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.996979 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.997012 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret podName:d003ff65-18de-4f6a-91b8-5dd4c2db3e12 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:20.49697828 +0000 UTC m=+3.043823386 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret") pod "global-pull-secret-syncer-wvsd9" (UID: "d003ff65-18de-4f6a-91b8-5dd4c2db3e12") : object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997020 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997133 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-socket-dir\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997169 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-conf-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997178 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-daemon-config\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997224 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-etc-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.997231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997238 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/849b9b0e-e00b-4793-aba1-38110bc88117-host\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997250 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-lib-modules\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997273 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-system-cni-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997277 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-node-log\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jxdj\" (UniqueName: \"kubernetes.io/projected/849b9b0e-e00b-4793-aba1-38110bc88117-kube-api-access-4jxdj\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997315 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-socket-dir-parent\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997328 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-run-netns\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997373 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-cni-multus\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.997379 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997413 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e6057259-93e1-4083-b7c0-863ea9fff32c-hosts-file\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:19.997421 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs podName:8a6cf69c-8067-47bf-b481-a1691e96d9ad nodeName:}" failed. No retries permitted until 2026-07-08 16:03:20.497407674 +0000 UTC m=+3.044252773 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs") pod "network-metrics-daemon-8s9kq" (UID: "8a6cf69c-8067-47bf-b481-a1691e96d9ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997455 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-var-lib-kubelet\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997458 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-etc-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997489 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-multus-certs\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997500 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-socket-dir-parent\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997509 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-lib-modules\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997520 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-etc-kubernetes\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998003 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997546 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-multus-certs\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997549 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-var-lib-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997556 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-systemd\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997594 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-os-release\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997597 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-etc-kubernetes\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997595 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-var-lib-openvswitch\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997735 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c18d608-ccd9-4075-9224-2b9a50cea4f7-agent-certs\") pod \"konnectivity-agent-zkbjr\" (UID: \"2c18d608-ccd9-4075-9224-2b9a50cea4f7\") " pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997786 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-os-release\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997783 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-slash\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997829 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-modprobe-d\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997872 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/849b9b0e-e00b-4793-aba1-38110bc88117-serviceca\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997897 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-cni-binary-copy\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-netns\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.997958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-cni-bin\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998011 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-netns\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998017 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-cnibin\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998043 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-systemd-units\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998094 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysconfig\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.998808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998119 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-kubernetes\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998146 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-kubelet\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-td7wb\" (UniqueName: \"kubernetes.io/projected/8a6cf69c-8067-47bf-b481-a1691e96d9ad-kube-api-access-td7wb\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998236 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-env-overrides\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998269 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zg7\" (UniqueName: \"kubernetes.io/projected/d05883b4-7ca2-49d8-80da-a2ba77d81548-kube-api-access-p7zg7\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998294 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-ovn\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998324 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998352 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovn-node-metrics-cert\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998376 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysctl-d\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998401 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9dn\" (UniqueName: \"kubernetes.io/projected/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-kube-api-access-ft9dn\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998440 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c18d608-ccd9-4075-9224-2b9a50cea4f7-konnectivity-ca\") pod \"konnectivity-agent-zkbjr\" (UID: \"2c18d608-ccd9-4075-9224-2b9a50cea4f7\") " pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998464 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-cni-binary-copy\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998522 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-sys-fs\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998558 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-kubelet\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998560 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-modprobe-d\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998467 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d05883b4-7ca2-49d8-80da-a2ba77d81548-sys-fs\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998596 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-run\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:19.999716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998622 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c48df59e-0f52-49f7-8803-7038efe0952f-etc-tuned\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998663 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-cnibin\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998695 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovnkube-script-lib\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998721 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-host\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-hostroot\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998857 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-hostroot\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998869 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-slash\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998879 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6057259-93e1-4083-b7c0-863ea9fff32c-tmp-dir\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-ovn\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvzdd\" (UniqueName: \"kubernetes.io/projected/ad541a88-86e4-4477-9db5-d361b00f1268-kube-api-access-zvzdd\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998944 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-log-socket\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998972 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-cni-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.998997 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32b8c356-9ba0-4e24-81e0-aab17229aab6-host-slash\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999023 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-kubelet\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-kubelet-config\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999093 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-dbus\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999117 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-sys\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999140 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-k8s-cni-cncf-io\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.000529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-systemd\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999193 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovnkube-config\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999216 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-phcks\" (UniqueName: \"kubernetes.io/projected/d5663b42-29a9-4cc9-9927-f401c43c45d7-kube-api-access-phcks\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999306 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-env-overrides\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999486 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-var-lib-cni-bin\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999856 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-dbus\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999866 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6057259-93e1-4083-b7c0-863ea9fff32c-tmp-dir\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999890 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/849b9b0e-e00b-4793-aba1-38110bc88117-serviceca\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999907 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-log-socket\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999941 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-kubelet\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999944 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-host-run-ovn-kubernetes\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999949 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-host\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999985 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-cnibin\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:19.999983 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-host-run-k8s-cni-cncf-io\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000023 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-run-systemd\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000077 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-kubernetes\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000116 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d5663b42-29a9-4cc9-9927-f401c43c45d7-systemd-units\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000163 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-run\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.001358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000188 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysconfig\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.002204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000328 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-sys\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.002204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000666 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovnkube-config\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.002204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000717 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c48df59e-0f52-49f7-8803-7038efe0952f-etc-sysctl-d\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.002204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000762 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-kubelet-config\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:20.002204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000805 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovnkube-script-lib\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.002204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.000876 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-multus-cni-dir\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.002204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.001217 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c48df59e-0f52-49f7-8803-7038efe0952f-tmp\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.002204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.001566 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c48df59e-0f52-49f7-8803-7038efe0952f-etc-tuned\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.002883 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.002824 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" event={"ID":"907e537dbcd976156b6c67c78c05344d","Type":"ContainerStarted","Data":"a9b6bfb5560792e4021e9b810a5bf303b6d46098cb67cc6486dbf22939e137d1"} Jul 08 16:03:20.003602 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.003555 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d5663b42-29a9-4cc9-9927-f401c43c45d7-ovn-node-metrics-cert\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.004119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.004096 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" event={"ID":"0f73b543c0aedc66b9e736106d082533","Type":"ContainerStarted","Data":"fb33398fb504c3620214bbcff554db16fbc29fa0a04eee153c80f3064e2bdd53"} Jul 08 16:03:20.010454 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.010433 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 16:03:20.010454 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.010457 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 16:03:20.010687 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.010471 2563 projected.go:196] Error preparing data for projected volume kube-api-access-vbwtc for pod openshift-network-diagnostics/network-check-target-gfhcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:20.010687 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.010555 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc podName:d4b8a21c-b467-4c86-9973-0e1945fc5a20 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:20.510538182 +0000 UTC m=+3.057383288 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwtc" (UniqueName: "kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc") pod "network-check-target-gfhcb" (UID: "d4b8a21c-b467-4c86-9973-0e1945fc5a20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:20.013844 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.013742 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhkc7\" (UniqueName: \"kubernetes.io/projected/c48df59e-0f52-49f7-8803-7038efe0952f-kube-api-access-jhkc7\") pod \"tuned-f5tfl\" (UID: \"c48df59e-0f52-49f7-8803-7038efe0952f\") " pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.014603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.014580 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jxdj\" (UniqueName: \"kubernetes.io/projected/849b9b0e-e00b-4793-aba1-38110bc88117-kube-api-access-4jxdj\") pod \"node-ca-27pnz\" (UID: \"849b9b0e-e00b-4793-aba1-38110bc88117\") " pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:20.015951 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.015908 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-td7wb\" (UniqueName: \"kubernetes.io/projected/8a6cf69c-8067-47bf-b481-a1691e96d9ad-kube-api-access-td7wb\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:20.016340 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.016314 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-phcks\" (UniqueName: \"kubernetes.io/projected/d5663b42-29a9-4cc9-9927-f401c43c45d7-kube-api-access-phcks\") pod \"ovnkube-node-g74wh\" (UID: \"d5663b42-29a9-4cc9-9927-f401c43c45d7\") " pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.016423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.016340 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zg7\" (UniqueName: \"kubernetes.io/projected/d05883b4-7ca2-49d8-80da-a2ba77d81548-kube-api-access-p7zg7\") pod \"aws-ebs-csi-driver-node-fs9p8\" (UID: \"d05883b4-7ca2-49d8-80da-a2ba77d81548\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:20.016714 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.016675 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9dn\" (UniqueName: \"kubernetes.io/projected/a0c81fb9-2d8d-4fa0-9916-c5f4150842e3-kube-api-access-ft9dn\") pod \"multus-pxnkf\" (UID: \"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3\") " pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.017107 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.017040 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg57v\" (UniqueName: \"kubernetes.io/projected/e6057259-93e1-4083-b7c0-863ea9fff32c-kube-api-access-mg57v\") pod \"node-resolver-jnp8q\" (UID: \"e6057259-93e1-4083-b7c0-863ea9fff32c\") " pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:20.099484 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099456 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbq62\" (UniqueName: \"kubernetes.io/projected/32b8c356-9ba0-4e24-81e0-aab17229aab6-kube-api-access-dbq62\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:20.099669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099494 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-os-release\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.099669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099513 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.099669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099541 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32b8c356-9ba0-4e24-81e0-aab17229aab6-iptables-alerter-script\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:20.099669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099618 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-os-release\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.099669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099664 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-system-cni-dir\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.099971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099716 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.099971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099748 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-system-cni-dir\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.099971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099762 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c18d608-ccd9-4075-9224-2b9a50cea4f7-agent-certs\") pod \"konnectivity-agent-zkbjr\" (UID: \"2c18d608-ccd9-4075-9224-2b9a50cea4f7\") " pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:20.099971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099799 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-cnibin\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.099971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099837 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c18d608-ccd9-4075-9224-2b9a50cea4f7-konnectivity-ca\") pod \"konnectivity-agent-zkbjr\" (UID: \"2c18d608-ccd9-4075-9224-2b9a50cea4f7\") " pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:20.099971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099874 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvzdd\" (UniqueName: \"kubernetes.io/projected/ad541a88-86e4-4477-9db5-d361b00f1268-kube-api-access-zvzdd\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.099971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099904 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32b8c356-9ba0-4e24-81e0-aab17229aab6-host-slash\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:20.100354 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.099957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-cni-binary-copy\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.100354 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.100354 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100166 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-cnibin\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.100354 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100171 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/32b8c356-9ba0-4e24-81e0-aab17229aab6-host-slash\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:20.100354 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100202 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/ad541a88-86e4-4477-9db5-d361b00f1268-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.100354 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100220 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/32b8c356-9ba0-4e24-81e0-aab17229aab6-iptables-alerter-script\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:20.100354 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100220 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.100354 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100220 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.100669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100639 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/ad541a88-86e4-4477-9db5-d361b00f1268-cni-binary-copy\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.100966 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.100907 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2c18d608-ccd9-4075-9224-2b9a50cea4f7-konnectivity-ca\") pod \"konnectivity-agent-zkbjr\" (UID: \"2c18d608-ccd9-4075-9224-2b9a50cea4f7\") " pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:20.103016 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.102798 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2c18d608-ccd9-4075-9224-2b9a50cea4f7-agent-certs\") pod \"konnectivity-agent-zkbjr\" (UID: \"2c18d608-ccd9-4075-9224-2b9a50cea4f7\") " pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:20.110844 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.110820 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbq62\" (UniqueName: \"kubernetes.io/projected/32b8c356-9ba0-4e24-81e0-aab17229aab6-kube-api-access-dbq62\") pod \"iptables-alerter-x9p9z\" (UID: \"32b8c356-9ba0-4e24-81e0-aab17229aab6\") " pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:20.117090 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.117020 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvzdd\" (UniqueName: \"kubernetes.io/projected/ad541a88-86e4-4477-9db5-d361b00f1268-kube-api-access-zvzdd\") pod \"multus-additional-cni-plugins-ztg8s\" (UID: \"ad541a88-86e4-4477-9db5-d361b00f1268\") " pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.185380 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.185350 2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 08 16:03:20.188747 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.188727 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" Jul 08 16:03:20.196741 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.196719 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" Jul 08 16:03:20.198670 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.198648 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd05883b4_7ca2_49d8_80da_a2ba77d81548.slice/crio-2b294cd22a5df3fc12f8cf8e024b54401a9bfeb1dab4ea49257e1f7a6ad7efd7 WatchSource:0}: Error finding container 2b294cd22a5df3fc12f8cf8e024b54401a9bfeb1dab4ea49257e1f7a6ad7efd7: Status 404 returned error can't find the container with id 2b294cd22a5df3fc12f8cf8e024b54401a9bfeb1dab4ea49257e1f7a6ad7efd7 Jul 08 16:03:20.203797 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.203779 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jnp8q" Jul 08 16:03:20.204113 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.204093 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc48df59e_0f52_49f7_8803_7038efe0952f.slice/crio-e6c8d1fc356800c39c3bc1ba087661f8a0f16a618a158d7a5c2d0c7c6b25e223 WatchSource:0}: Error finding container e6c8d1fc356800c39c3bc1ba087661f8a0f16a618a158d7a5c2d0c7c6b25e223: Status 404 returned error can't find the container with id e6c8d1fc356800c39c3bc1ba087661f8a0f16a618a158d7a5c2d0c7c6b25e223 Jul 08 16:03:20.209880 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.209864 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-27pnz" Jul 08 16:03:20.211215 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.211190 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6057259_93e1_4083_b7c0_863ea9fff32c.slice/crio-7ebf6e4ca87627aab18d0396ec23b4151120d97a39a58c8c7fd4fb83febf8df8 WatchSource:0}: Error finding container 7ebf6e4ca87627aab18d0396ec23b4151120d97a39a58c8c7fd4fb83febf8df8: Status 404 returned error can't find the container with id 7ebf6e4ca87627aab18d0396ec23b4151120d97a39a58c8c7fd4fb83febf8df8 Jul 08 16:03:20.218559 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.218542 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849b9b0e_e00b_4793_aba1_38110bc88117.slice/crio-6d2ab3f7e9794999daa2ff962e7e6bccd1d8b626f15ed4a1547cd093b021e9af WatchSource:0}: Error finding container 6d2ab3f7e9794999daa2ff962e7e6bccd1d8b626f15ed4a1547cd093b021e9af: Status 404 returned error can't find the container with id 6d2ab3f7e9794999daa2ff962e7e6bccd1d8b626f15ed4a1547cd093b021e9af Jul 08 16:03:20.219111 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.219098 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-pxnkf" Jul 08 16:03:20.226057 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.226038 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0c81fb9_2d8d_4fa0_9916_c5f4150842e3.slice/crio-3cfa96374668d459fb9ef4b752df517e3414ba8793973ae12cad7d45b6299bed WatchSource:0}: Error finding container 3cfa96374668d459fb9ef4b752df517e3414ba8793973ae12cad7d45b6299bed: Status 404 returned error can't find the container with id 3cfa96374668d459fb9ef4b752df517e3414ba8793973ae12cad7d45b6299bed Jul 08 16:03:20.226856 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.226843 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:20.233473 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.233453 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5663b42_29a9_4cc9_9927_f401c43c45d7.slice/crio-98ecd696f48f73646401b3cf51c804e27786affd7b0f668f3af983efe843222c WatchSource:0}: Error finding container 98ecd696f48f73646401b3cf51c804e27786affd7b0f668f3af983efe843222c: Status 404 returned error can't find the container with id 98ecd696f48f73646401b3cf51c804e27786affd7b0f668f3af983efe843222c Jul 08 16:03:20.237245 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.237227 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-x9p9z" Jul 08 16:03:20.244013 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.243996 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:20.244255 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.244235 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b8c356_9ba0_4e24_81e0_aab17229aab6.slice/crio-73bf56aeb341c7de3912a65780ce056084152e70c564359a04cf2a71c2902aa9 WatchSource:0}: Error finding container 73bf56aeb341c7de3912a65780ce056084152e70c564359a04cf2a71c2902aa9: Status 404 returned error can't find the container with id 73bf56aeb341c7de3912a65780ce056084152e70c564359a04cf2a71c2902aa9 Jul 08 16:03:20.250338 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.250321 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" Jul 08 16:03:20.254051 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.254031 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c18d608_ccd9_4075_9224_2b9a50cea4f7.slice/crio-0dd42a23377d79205df6619a65e59f9c0db49462dd1bb863d855af2e7ad0b620 WatchSource:0}: Error finding container 0dd42a23377d79205df6619a65e59f9c0db49462dd1bb863d855af2e7ad0b620: Status 404 returned error can't find the container with id 0dd42a23377d79205df6619a65e59f9c0db49462dd1bb863d855af2e7ad0b620 Jul 08 16:03:20.258736 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:20.258715 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad541a88_86e4_4477_9db5_d361b00f1268.slice/crio-c2cdf6f0ffb9d4ba77a2a3af846d03058847891e04e03bda85318225f90ccb4e WatchSource:0}: Error finding container c2cdf6f0ffb9d4ba77a2a3af846d03058847891e04e03bda85318225f90ccb4e: Status 404 returned error can't find the container with id c2cdf6f0ffb9d4ba77a2a3af846d03058847891e04e03bda85318225f90ccb4e Jul 08 16:03:20.502477 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.502311 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:20.502477 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.502363 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:20.502677 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.502520 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:20.502677 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.502595 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs podName:8a6cf69c-8067-47bf-b481-a1691e96d9ad nodeName:}" failed. No retries permitted until 2026-07-08 16:03:21.502575701 +0000 UTC m=+4.049420816 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs") pod "network-metrics-daemon-8s9kq" (UID: "8a6cf69c-8067-47bf-b481-a1691e96d9ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:20.502677 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.502598 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:20.502677 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.502661 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret podName:d003ff65-18de-4f6a-91b8-5dd4c2db3e12 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:21.502643328 +0000 UTC m=+4.049488443 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret") pod "global-pull-secret-syncer-wvsd9" (UID: "d003ff65-18de-4f6a-91b8-5dd4c2db3e12") : object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:20.602753 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.602701 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:20.602953 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.602932 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 16:03:20.603018 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.602960 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 16:03:20.603018 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.602972 2563 projected.go:196] Error preparing data for projected volume kube-api-access-vbwtc for pod openshift-network-diagnostics/network-check-target-gfhcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:20.603153 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:20.603036 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc podName:d4b8a21c-b467-4c86-9973-0e1945fc5a20 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:21.60301688 +0000 UTC m=+4.149861994 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwtc" (UniqueName: "kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc") pod "network-check-target-gfhcb" (UID: "d4b8a21c-b467-4c86-9973-0e1945fc5a20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:20.942970 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.942872 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 15:58:19 +0000 UTC" deadline="2028-01-22 17:57:09.050665959 +0000 UTC" Jul 08 16:03:20.942970 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:20.942921 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13513h53m48.107749057s" Jul 08 16:03:21.008204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.008167 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkbjr" event={"ID":"2c18d608-ccd9-4075-9224-2b9a50cea4f7","Type":"ContainerStarted","Data":"0dd42a23377d79205df6619a65e59f9c0db49462dd1bb863d855af2e7ad0b620"} Jul 08 16:03:21.010262 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.010235 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"98ecd696f48f73646401b3cf51c804e27786affd7b0f668f3af983efe843222c"} Jul 08 16:03:21.012351 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.012316 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-27pnz" event={"ID":"849b9b0e-e00b-4793-aba1-38110bc88117","Type":"ContainerStarted","Data":"6d2ab3f7e9794999daa2ff962e7e6bccd1d8b626f15ed4a1547cd093b021e9af"} Jul 08 16:03:21.014107 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.014084 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jnp8q" event={"ID":"e6057259-93e1-4083-b7c0-863ea9fff32c","Type":"ContainerStarted","Data":"7ebf6e4ca87627aab18d0396ec23b4151120d97a39a58c8c7fd4fb83febf8df8"} Jul 08 16:03:21.015612 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.015588 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" event={"ID":"d05883b4-7ca2-49d8-80da-a2ba77d81548","Type":"ContainerStarted","Data":"2b294cd22a5df3fc12f8cf8e024b54401a9bfeb1dab4ea49257e1f7a6ad7efd7"} Jul 08 16:03:21.017477 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.017453 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerStarted","Data":"c2cdf6f0ffb9d4ba77a2a3af846d03058847891e04e03bda85318225f90ccb4e"} Jul 08 16:03:21.019236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.019212 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x9p9z" event={"ID":"32b8c356-9ba0-4e24-81e0-aab17229aab6","Type":"ContainerStarted","Data":"73bf56aeb341c7de3912a65780ce056084152e70c564359a04cf2a71c2902aa9"} Jul 08 16:03:21.020876 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.020851 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxnkf" event={"ID":"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3","Type":"ContainerStarted","Data":"3cfa96374668d459fb9ef4b752df517e3414ba8793973ae12cad7d45b6299bed"} Jul 08 16:03:21.022870 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.022829 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" event={"ID":"c48df59e-0f52-49f7-8803-7038efe0952f","Type":"ContainerStarted","Data":"e6c8d1fc356800c39c3bc1ba087661f8a0f16a618a158d7a5c2d0c7c6b25e223"} Jul 08 16:03:21.510108 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.510058 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:21.510292 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.510134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:21.510292 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:21.510204 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:21.510292 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:21.510212 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:21.510292 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:21.510284 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs podName:8a6cf69c-8067-47bf-b481-a1691e96d9ad nodeName:}" failed. No retries permitted until 2026-07-08 16:03:23.510264843 +0000 UTC m=+6.057109951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs") pod "network-metrics-daemon-8s9kq" (UID: "8a6cf69c-8067-47bf-b481-a1691e96d9ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:21.510512 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:21.510311 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret podName:d003ff65-18de-4f6a-91b8-5dd4c2db3e12 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:23.510298169 +0000 UTC m=+6.057143275 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret") pod "global-pull-secret-syncer-wvsd9" (UID: "d003ff65-18de-4f6a-91b8-5dd4c2db3e12") : object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:21.611255 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.611201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:21.611467 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:21.611413 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 16:03:21.611467 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:21.611438 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 16:03:21.611467 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:21.611451 2563 projected.go:196] Error preparing data for projected volume kube-api-access-vbwtc for pod openshift-network-diagnostics/network-check-target-gfhcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:21.611647 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:21.611507 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc podName:d4b8a21c-b467-4c86-9973-0e1945fc5a20 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:23.611489835 +0000 UTC m=+6.158334930 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwtc" (UniqueName: "kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc") pod "network-check-target-gfhcb" (UID: "d4b8a21c-b467-4c86-9973-0e1945fc5a20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:21.943772 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.943686 2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-07 15:58:19 +0000 UTC" deadline="2028-01-15 12:19:38.455492127 +0000 UTC" Jul 08 16:03:21.943772 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:21.943728 2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13340h16m16.511768236s" Jul 08 16:03:22.007499 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:22.007474 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:22.007650 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:22.007603 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:22.008039 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:22.008005 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:22.008288 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:22.008145 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:22.008288 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:22.008199 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:22.008288 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:22.008275 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:23.529393 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:23.529356 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:23.529860 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:23.529414 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:23.529860 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.529585 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:23.529860 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.529642 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret podName:d003ff65-18de-4f6a-91b8-5dd4c2db3e12 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:27.529625855 +0000 UTC m=+10.076470949 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret") pod "global-pull-secret-syncer-wvsd9" (UID: "d003ff65-18de-4f6a-91b8-5dd4c2db3e12") : object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:23.530058 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.530035 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:23.530147 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.530114 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs podName:8a6cf69c-8067-47bf-b481-a1691e96d9ad nodeName:}" failed. No retries permitted until 2026-07-08 16:03:27.530096554 +0000 UTC m=+10.076941647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs") pod "network-metrics-daemon-8s9kq" (UID: "8a6cf69c-8067-47bf-b481-a1691e96d9ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:23.630593 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:23.630557 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:23.631174 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.630744 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 16:03:23.631174 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.630770 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 16:03:23.631174 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.630783 2563 projected.go:196] Error preparing data for projected volume kube-api-access-vbwtc for pod openshift-network-diagnostics/network-check-target-gfhcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:23.631174 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.630844 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc podName:d4b8a21c-b467-4c86-9973-0e1945fc5a20 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:27.630825343 +0000 UTC m=+10.177670441 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwtc" (UniqueName: "kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc") pod "network-check-target-gfhcb" (UID: "d4b8a21c-b467-4c86-9973-0e1945fc5a20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:23.997681 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:23.997186 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:23.997681 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:23.997211 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:23.997681 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:23.997192 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:23.997681 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.997310 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:23.997681 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.997416 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:23.997681 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:23.997508 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:25.999142 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:25.999120 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:25.999548 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:25.999235 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:25.999623 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:25.999577 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:25.999825 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:25.999669 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:25.999825 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:25.999725 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:25.999825 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:25.999773 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:27.562384 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:27.562342 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:27.562862 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:27.562398 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:27.562862 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.562509 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:27.562862 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.562525 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:27.562862 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.562582 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret podName:d003ff65-18de-4f6a-91b8-5dd4c2db3e12 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:35.562564592 +0000 UTC m=+18.109409688 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret") pod "global-pull-secret-syncer-wvsd9" (UID: "d003ff65-18de-4f6a-91b8-5dd4c2db3e12") : object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:27.562862 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.562603 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs podName:8a6cf69c-8067-47bf-b481-a1691e96d9ad nodeName:}" failed. No retries permitted until 2026-07-08 16:03:35.562593357 +0000 UTC m=+18.109438454 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs") pod "network-metrics-daemon-8s9kq" (UID: "8a6cf69c-8067-47bf-b481-a1691e96d9ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:27.663703 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:27.663623 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:27.663897 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.663780 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 16:03:27.663897 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.663799 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 16:03:27.663897 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.663811 2563 projected.go:196] Error preparing data for projected volume kube-api-access-vbwtc for pod openshift-network-diagnostics/network-check-target-gfhcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:27.663897 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.663873 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc podName:d4b8a21c-b467-4c86-9973-0e1945fc5a20 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:35.663855159 +0000 UTC m=+18.210700269 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwtc" (UniqueName: "kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc") pod "network-check-target-gfhcb" (UID: "d4b8a21c-b467-4c86-9973-0e1945fc5a20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:27.998366 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:27.997806 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:27.998366 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.997974 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:27.998366 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:27.998053 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:27.998366 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:27.998161 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:27.998366 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.998161 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:27.998366 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:27.998277 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:29.996179 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:29.996107 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:29.996644 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:29.996246 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:29.996644 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:29.996107 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:29.996644 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:29.996348 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:29.996644 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:29.996411 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:29.996851 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:29.996613 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:31.996606 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:31.996387 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:31.997031 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:31.996387 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:31.997031 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:31.996700 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:31.997031 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:31.996436 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:31.997031 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:31.996757 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:31.997031 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:31.996891 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:33.996372 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:33.996337 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:33.996759 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:33.996341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:33.996759 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:33.996489 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:33.996759 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:33.996344 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:33.996759 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:33.996535 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:33.996759 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:33.996588 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:35.621099 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:35.621050 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:35.621612 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:35.621111 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:35.621612 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.621195 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:35.621612 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.621241 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:35.621612 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.621274 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs podName:8a6cf69c-8067-47bf-b481-a1691e96d9ad nodeName:}" failed. No retries permitted until 2026-07-08 16:03:51.621252857 +0000 UTC m=+34.168097969 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs") pod "network-metrics-daemon-8s9kq" (UID: "8a6cf69c-8067-47bf-b481-a1691e96d9ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:35.621612 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.621294 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret podName:d003ff65-18de-4f6a-91b8-5dd4c2db3e12 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:51.621285285 +0000 UTC m=+34.168130382 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret") pod "global-pull-secret-syncer-wvsd9" (UID: "d003ff65-18de-4f6a-91b8-5dd4c2db3e12") : object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:35.722017 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:35.721984 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:35.722203 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.722142 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 16:03:35.722203 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.722163 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 16:03:35.722203 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.722174 2563 projected.go:196] Error preparing data for projected volume kube-api-access-vbwtc for pod openshift-network-diagnostics/network-check-target-gfhcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:35.722332 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.722228 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc podName:d4b8a21c-b467-4c86-9973-0e1945fc5a20 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:51.722213411 +0000 UTC m=+34.269058506 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwtc" (UniqueName: "kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc") pod "network-check-target-gfhcb" (UID: "d4b8a21c-b467-4c86-9973-0e1945fc5a20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:35.996487 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:35.996443 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:35.996647 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:35.996443 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:35.996647 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.996574 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:35.996647 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:35.996597 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:35.996780 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.996653 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:35.996780 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:35.996761 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:37.997706 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:37.997674 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:37.998187 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:37.997804 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:37.998187 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:37.997888 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:37.998187 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:37.998087 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:37.998187 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:37.998094 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:37.998390 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:37.998192 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:39.996814 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:39.996424 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:39.997265 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:39.996838 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:39.997265 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:39.996550 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:39.997265 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:39.996960 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:39.997265 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:39.996424 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:39.997265 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:39.997038 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:40.066376 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:40.066333 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" event={"ID":"0f73b543c0aedc66b9e736106d082533","Type":"ContainerStarted","Data":"3ae5e9e49f9c89e8b67e3f882b8761fa99d4b7c650b43881c1d82c54705706ee"} Jul 08 16:03:40.069274 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:40.068692 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-pxnkf" event={"ID":"a0c81fb9-2d8d-4fa0-9916-c5f4150842e3","Type":"ContainerStarted","Data":"3e0c9bd3858f499e6a8c905ed322ca40dfed28250f065ecf427602a7adc7824b"} Jul 08 16:03:40.071023 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:40.071000 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" event={"ID":"c48df59e-0f52-49f7-8803-7038efe0952f","Type":"ContainerStarted","Data":"18bcf0fa63a0f9bb53f3293ccf02df1619941c592a044c0c20383c97e93d3467"} Jul 08 16:03:40.072791 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:40.072759 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"ca0d4ca61d8cfde378d647a219af3cae9ba0c66863156973d300c5c1001c23f5"} Jul 08 16:03:40.083631 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:40.083352 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-121.ec2.internal" podStartSLOduration=21.083339068 podStartE2EDuration="21.083339068s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:40.083325885 +0000 UTC m=+22.630170998" watchObservedRunningTime="2026-07-08 16:03:40.083339068 +0000 UTC m=+22.630184181" Jul 08 16:03:40.121599 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:40.121553 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f5tfl" podStartSLOduration=2.973063839 podStartE2EDuration="22.121536159s" totalImagesPullingTime="19.14847232s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:40.10177628 +0000 UTC m=+22.648621395" watchObservedRunningTime="2026-07-08 16:03:40.121536159 +0000 UTC m=+22.668381266" Jul 08 16:03:40.121858 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:40.121836 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-pxnkf" podStartSLOduration=2.958803252 podStartE2EDuration="22.1218314s" totalImagesPullingTime="19.163028148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:40.121673423 +0000 UTC m=+22.668518520" watchObservedRunningTime="2026-07-08 16:03:40.1218314 +0000 UTC m=+22.668676514" Jul 08 16:03:41.076585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.076310 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" event={"ID":"d05883b4-7ca2-49d8-80da-a2ba77d81548","Type":"ContainerStarted","Data":"6270a4e1179abed0c097d01bb42dbcbad6fe721570181f19e7db93872abb4253"} Jul 08 16:03:41.077829 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.077805 2563 generic.go:356] "Generic (PLEG): container finished" podID="907e537dbcd976156b6c67c78c05344d" containerID="393c7ed7ff71c21430fb726d87ee658602a1f4ad4a41f6bf1f349d1d313416f4" exitCode=0 Jul 08 16:03:41.077914 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.077875 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" event={"ID":"907e537dbcd976156b6c67c78c05344d","Type":"ContainerDied","Data":"393c7ed7ff71c21430fb726d87ee658602a1f4ad4a41f6bf1f349d1d313416f4"} Jul 08 16:03:41.079402 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.079377 2563 generic.go:356] "Generic (PLEG): container finished" podID="ad541a88-86e4-4477-9db5-d361b00f1268" containerID="e067e92a84109f9ab9088de794032cdf8e58fbc19f95cd5ad83df921968a45ab" exitCode=0 Jul 08 16:03:41.079485 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.079448 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerDied","Data":"e067e92a84109f9ab9088de794032cdf8e58fbc19f95cd5ad83df921968a45ab"} Jul 08 16:03:41.080876 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.080856 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-x9p9z" event={"ID":"32b8c356-9ba0-4e24-81e0-aab17229aab6","Type":"ContainerStarted","Data":"8a99867c8468398c4f027186e5981a240d4594a8979b97e3d5877e3ee6cb2757"} Jul 08 16:03:41.082478 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.082446 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zkbjr" event={"ID":"2c18d608-ccd9-4075-9224-2b9a50cea4f7","Type":"ContainerStarted","Data":"4211a48c92535705a62074bed278ac2c2f6474f725522009ba07f26daefd2524"} Jul 08 16:03:41.085282 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.085265 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"7b0a767f93d261959773bf9445538623438d521952a26d2c69f6b1dfbd1e7184"} Jul 08 16:03:41.085366 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.085325 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"55de2bfc10d464a2d7c7d11f382b14bf127350d01355dd589274465953571f1a"} Jul 08 16:03:41.085366 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.085356 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"3c4aef8717aad60c608d00c2a0743c8b7df8c257b8230a076708874325b8791c"} Jul 08 16:03:41.085451 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.085388 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"36bf04fc08f79367988c8a2eb4520f350b0e176fc9bf66b09acf76fc73793ea6"} Jul 08 16:03:41.085451 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.085425 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"1d16e36195ce9559156b1d7779c69cd7378427cb0872f6ec269bef03aa49ee21"} Jul 08 16:03:41.086452 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.086434 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-27pnz" event={"ID":"849b9b0e-e00b-4793-aba1-38110bc88117","Type":"ContainerStarted","Data":"f5a0d206526180192093aa406105c81dbc79e01eb99828637da0334f0402a674"} Jul 08 16:03:41.087643 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.087627 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jnp8q" event={"ID":"e6057259-93e1-4083-b7c0-863ea9fff32c","Type":"ContainerStarted","Data":"bb650e914ce11041acae6034670339dda46dd41fb6f8b2d76f7d352bdb2dc50e"} Jul 08 16:03:41.112426 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.112388 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-x9p9z" podStartSLOduration=8.50829787 podStartE2EDuration="23.112374072s" totalImagesPullingTime="14.604076202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:41.112174791 +0000 UTC m=+23.659019902" watchObservedRunningTime="2026-07-08 16:03:41.112374072 +0000 UTC m=+23.659219209" Jul 08 16:03:41.130818 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.130780 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zkbjr" podStartSLOduration=4.008142315 podStartE2EDuration="23.130766974s" totalImagesPullingTime="19.122624659s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:41.130555707 +0000 UTC m=+23.677400822" watchObservedRunningTime="2026-07-08 16:03:41.130766974 +0000 UTC m=+23.677612087" Jul 08 16:03:41.149004 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.148958 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jnp8q" podStartSLOduration=8.51377842 podStartE2EDuration="23.14894118s" totalImagesPullingTime="14.63516276s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:41.148718675 +0000 UTC m=+23.695563790" watchObservedRunningTime="2026-07-08 16:03:41.14894118 +0000 UTC m=+23.695786298" Jul 08 16:03:41.195425 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.195383 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-27pnz" podStartSLOduration=4.037246964 podStartE2EDuration="23.195367055s" totalImagesPullingTime="19.158120091s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:41.166258064 +0000 UTC m=+23.713103181" watchObservedRunningTime="2026-07-08 16:03:41.195367055 +0000 UTC m=+23.742212248" Jul 08 16:03:41.996926 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.996886 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:41.997307 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.996891 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:41.997307 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:41.997020 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:41.997307 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:41.997015 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:41.997307 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:41.997130 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:41.997307 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:41.997219 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:42.092482 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:42.092441 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" event={"ID":"907e537dbcd976156b6c67c78c05344d","Type":"ContainerStarted","Data":"638f71f7b200ddf5f240f94b77d2bca08d43f8ec1df80aad49c6a46ef1525836"} Jul 08 16:03:42.110890 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:42.110838 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-121.ec2.internal" podStartSLOduration=23.110818404 podStartE2EDuration="23.110818404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:42.110265413 +0000 UTC m=+24.657110527" watchObservedRunningTime="2026-07-08 16:03:42.110818404 +0000 UTC m=+24.657663520" Jul 08 16:03:42.443505 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:42.443477 2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 08 16:03:42.950286 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:42.950181 2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-08T16:03:42.443500875Z","UUID":"4d6a836c-125c-4682-83ad-031c1fed0244","Handler":null,"Name":"","Endpoint":""} Jul 08 16:03:42.951784 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:42.951760 2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 08 16:03:42.951967 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:42.951805 2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 08 16:03:43.099146 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:43.099053 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"ce693c7155f40bfcc3cb08f1341c03921ab9ca1ab1494b0c4d3bcc4ba58e2586"} Jul 08 16:03:43.101207 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:43.101182 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" event={"ID":"d05883b4-7ca2-49d8-80da-a2ba77d81548","Type":"ContainerStarted","Data":"f8ababfe0019771769c7c142e417b0c254dc6b9430673b756630b0bda1dd00ce"} Jul 08 16:03:43.572020 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:43.571992 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:43.572669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:43.572648 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:43.996240 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:43.996211 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:43.996343 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:43.996237 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:43.996343 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:43.996237 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:43.996343 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:43.996319 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:43.996475 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:43.996442 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:43.996579 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:43.996549 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:44.106380 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:44.106137 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" event={"ID":"d05883b4-7ca2-49d8-80da-a2ba77d81548","Type":"ContainerStarted","Data":"21929bc569cbca4cf35ee698d06c8de9c3a478fbf943d169cf05414d99a873b1"} Jul 08 16:03:44.106808 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:44.106427 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:44.106918 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:44.106900 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zkbjr" Jul 08 16:03:44.127930 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:44.127888 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fs9p8" podStartSLOduration=3.597694574 podStartE2EDuration="26.127874767s" totalImagesPullingTime="22.530180193s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:44.126177152 +0000 UTC m=+26.673022268" watchObservedRunningTime="2026-07-08 16:03:44.127874767 +0000 UTC m=+26.674719881" Jul 08 16:03:45.996770 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:45.996736 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:45.996770 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:45.996763 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:45.997634 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:45.996850 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:45.997634 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:45.996892 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:45.997634 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:45.996972 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:45.997634 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:45.997098 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:46.114595 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:46.114560 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" event={"ID":"d5663b42-29a9-4cc9-9927-f401c43c45d7","Type":"ContainerStarted","Data":"254477c95660d2a5b15c5ff6f0b7d97f2124bc8a2a8ddfcd5bbb33896cc5c396"} Jul 08 16:03:46.114892 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:46.114870 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:46.114987 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:46.114935 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:46.131188 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:46.131157 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:46.147898 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:46.147850 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" podStartSLOduration=8.467719871 podStartE2EDuration="28.147833981s" totalImagesPullingTime="19.68011411s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:46.147270518 +0000 UTC m=+28.694115633" watchObservedRunningTime="2026-07-08 16:03:46.147833981 +0000 UTC m=+28.694679095" Jul 08 16:03:47.117480 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:47.117453 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:47.134723 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:47.134513 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gfhcb"] Jul 08 16:03:47.134872 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:47.134824 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:47.134958 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:47.134931 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:47.135597 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:47.135572 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:03:47.137821 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:47.137801 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8s9kq"] Jul 08 16:03:47.137931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:47.137892 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:47.138025 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:47.138006 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:47.142097 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:47.142050 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wvsd9"] Jul 08 16:03:47.142194 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:47.142157 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:47.142283 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:47.142246 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:48.120376 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:48.120341 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerStarted","Data":"14f7962e4df5374f52ff4031f9c0b38028c34041166f7183016f2e7e38f0d472"} Jul 08 16:03:48.996754 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:48.996716 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:48.996931 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:48.996826 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:48.996931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:48.996716 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:48.996931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:48.996716 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:48.997097 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:48.996945 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:48.997097 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:48.996981 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:49.123963 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:49.123936 2563 generic.go:356] "Generic (PLEG): container finished" podID="ad541a88-86e4-4477-9db5-d361b00f1268" containerID="14f7962e4df5374f52ff4031f9c0b38028c34041166f7183016f2e7e38f0d472" exitCode=0 Jul 08 16:03:49.124312 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:49.124013 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerDied","Data":"14f7962e4df5374f52ff4031f9c0b38028c34041166f7183016f2e7e38f0d472"} Jul 08 16:03:50.128217 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:50.128188 2563 generic.go:356] "Generic (PLEG): container finished" podID="ad541a88-86e4-4477-9db5-d361b00f1268" containerID="288e5218a493827728984d0c13fe434f2ab57cfc814af92585985b89cdd03fdb" exitCode=0 Jul 08 16:03:50.128594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:50.128235 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerDied","Data":"288e5218a493827728984d0c13fe434f2ab57cfc814af92585985b89cdd03fdb"} Jul 08 16:03:50.996103 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:50.996057 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:50.996242 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:50.996112 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:50.996242 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:50.996122 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:50.996242 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:50.996194 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-wvsd9" podUID="d003ff65-18de-4f6a-91b8-5dd4c2db3e12" Jul 08 16:03:50.996357 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:50.996302 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gfhcb" podUID="d4b8a21c-b467-4c86-9973-0e1945fc5a20" Jul 08 16:03:50.996388 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:50.996371 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8s9kq" podUID="8a6cf69c-8067-47bf-b481-a1691e96d9ad" Jul 08 16:03:51.651702 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:51.651629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:51.651702 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:51.651665 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:51.652083 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:51.651767 2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:51.652083 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:51.651768 2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:51.652083 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:51.651814 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret podName:d003ff65-18de-4f6a-91b8-5dd4c2db3e12 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:23.651802673 +0000 UTC m=+66.198647765 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret") pod "global-pull-secret-syncer-wvsd9" (UID: "d003ff65-18de-4f6a-91b8-5dd4c2db3e12") : object "kube-system"/"original-pull-secret" not registered Jul 08 16:03:51.652083 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:51.651828 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs podName:8a6cf69c-8067-47bf-b481-a1691e96d9ad nodeName:}" failed. No retries permitted until 2026-07-08 16:04:23.651821927 +0000 UTC m=+66.198667019 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs") pod "network-metrics-daemon-8s9kq" (UID: "8a6cf69c-8067-47bf-b481-a1691e96d9ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 08 16:03:51.752782 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:51.752747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:51.752937 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:51.752906 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 08 16:03:51.752975 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:51.752942 2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 08 16:03:51.752975 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:51.752953 2563 projected.go:196] Error preparing data for projected volume kube-api-access-vbwtc for pod openshift-network-diagnostics/network-check-target-gfhcb: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:51.753039 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:51.753002 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc podName:d4b8a21c-b467-4c86-9973-0e1945fc5a20 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:23.752987625 +0000 UTC m=+66.299832717 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vbwtc" (UniqueName: "kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc") pod "network-check-target-gfhcb" (UID: "d4b8a21c-b467-4c86-9973-0e1945fc5a20") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 08 16:03:52.134882 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.134848 2563 generic.go:356] "Generic (PLEG): container finished" podID="ad541a88-86e4-4477-9db5-d361b00f1268" containerID="e9b16de190d846de6bf7a5cedbecb3dabdd6d24a3c11d4ee17ecfa81a896709b" exitCode=0 Jul 08 16:03:52.135117 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.134899 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerDied","Data":"e9b16de190d846de6bf7a5cedbecb3dabdd6d24a3c11d4ee17ecfa81a896709b"} Jul 08 16:03:52.686438 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.686265 2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-121.ec2.internal" event="NodeReady" Jul 08 16:03:52.686862 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.686554 2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 08 16:03:52.740638 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.740605 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d67db9954-tkv9c"] Jul 08 16:03:52.766128 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.766096 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf"] Jul 08 16:03:52.766307 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.766209 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.774782 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.774735 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 08 16:03:52.775734 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.775495 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 08 16:03:52.775734 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.775585 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hjrqc\"" Jul 08 16:03:52.775734 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.775637 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 08 16:03:52.779477 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.779459 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 08 16:03:52.792716 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.792690 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5"] Jul 08 16:03:52.792871 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.792810 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf" Jul 08 16:03:52.795605 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.795583 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 08 16:03:52.795995 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.795976 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-k858w\"" Jul 08 16:03:52.796056 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.795976 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 08 16:03:52.816872 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.816847 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wp8nf"] Jul 08 16:03:52.817017 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.816961 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:52.819774 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.819756 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 08 16:03:52.820033 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.820019 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lwjvj\"" Jul 08 16:03:52.820101 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.820044 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 08 16:03:52.821489 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.821473 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 08 16:03:52.821581 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.821476 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 08 16:03:52.842542 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.842518 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777"] Jul 08 16:03:52.842646 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.842632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:52.845932 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.845917 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 08 16:03:52.846213 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.846074 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 08 16:03:52.846213 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.846145 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-lbqws\"" Jul 08 16:03:52.846213 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.846171 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 08 16:03:52.847362 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.847344 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 08 16:03:52.855670 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.855652 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 08 16:03:52.862129 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.862110 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-trusted-ca\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.862220 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.862154 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-installation-pull-secrets\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.862220 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.862213 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-bound-sa-token\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.862318 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.862247 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.862318 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.862270 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32c32588-d44b-4083-8fbd-85f47cf30ea5-ca-trust-extracted\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.862421 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.862337 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2z5b\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-kube-api-access-h2z5b\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.862421 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.862373 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-image-registry-private-configuration\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.862421 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.862398 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-certificates\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.867084 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.867053 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-75f96688f5-4vp7w"] Jul 08 16:03:52.867193 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.867182 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:52.870016 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.869997 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 08 16:03:52.870138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.870026 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 08 16:03:52.870138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.870035 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 08 16:03:52.870138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.870042 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 08 16:03:52.870138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.870043 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-chs9m\"" Jul 08 16:03:52.890921 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.890899 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv"] Jul 08 16:03:52.891025 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.891011 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:52.894987 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.894972 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 08 16:03:52.895460 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.895443 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gp2gw\"" Jul 08 16:03:52.895567 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.895551 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 08 16:03:52.896106 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.896080 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 08 16:03:52.896384 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.896166 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 08 16:03:52.896384 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.896178 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 08 16:03:52.896551 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.896209 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 08 16:03:52.914419 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.914401 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf"] Jul 08 16:03:52.914521 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.914435 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d67db9954-tkv9c"] Jul 08 16:03:52.914521 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.914452 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf"] Jul 08 16:03:52.914521 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.914471 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:52.917298 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.917283 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 08 16:03:52.917432 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.917419 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 08 16:03:52.917551 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.917535 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 08 16:03:52.917617 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.917607 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h58rq\"" Jul 08 16:03:52.937555 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.937492 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx"] Jul 08 16:03:52.937654 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.937611 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:52.940831 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.940807 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 08 16:03:52.940945 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.940876 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 08 16:03:52.940945 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.940876 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-l5qrz\"" Jul 08 16:03:52.941045 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.940876 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 08 16:03:52.941156 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.941144 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 08 16:03:52.958587 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.958563 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-vqmfn"] Jul 08 16:03:52.958693 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.958682 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:52.961595 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.961576 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 08 16:03:52.961704 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.961635 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vz8vc\"" Jul 08 16:03:52.961770 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.961723 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 08 16:03:52.962710 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.962689 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-image-registry-private-configuration\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.962772 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.962726 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-certificates\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.962772 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.962755 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x524c\" (UniqueName: \"kubernetes.io/projected/474c28fe-7e30-4d21-a705-1fc25a2ee18f-kube-api-access-x524c\") pod \"volume-data-source-validator-5d456fdbcd-qslqf\" (UID: \"474c28fe-7e30-4d21-a705-1fc25a2ee18f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf" Jul 08 16:03:52.962855 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.962839 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:52.962908 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.962889 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-trusted-ca\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.962958 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.962908 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhvlh\" (UniqueName: \"kubernetes.io/projected/1b73f265-f5e7-4aaa-a960-ea10101d0d55-kube-api-access-lhvlh\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:52.963007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.962958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-installation-pull-secrets\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.963007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.962988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-bound-sa-token\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.963119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963004 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0721a8f-4745-41d6-944c-d4ab9a293612-config\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:52.963119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963039 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.963119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963082 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b73f265-f5e7-4aaa-a960-ea10101d0d55-config\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:52.963119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963103 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b73f265-f5e7-4aaa-a960-ea10101d0d55-serving-cert\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:52.963314 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963131 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32c32588-d44b-4083-8fbd-85f47cf30ea5-ca-trust-extracted\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.963314 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963158 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b73f265-f5e7-4aaa-a960-ea10101d0d55-trusted-ca\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:52.963314 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963181 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvb8r\" (UniqueName: \"kubernetes.io/projected/d0721a8f-4745-41d6-944c-d4ab9a293612-kube-api-access-vvb8r\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:52.963314 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963207 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcsjz\" (UniqueName: \"kubernetes.io/projected/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-kube-api-access-fcsjz\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:52.963314 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963234 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0721a8f-4745-41d6-944c-d4ab9a293612-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:52.963314 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963266 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2z5b\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-kube-api-access-h2z5b\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.963314 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963289 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:52.963669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963391 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-certificates\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.963669 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:52.963498 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 16:03:52.963669 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:52.963559 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d67db9954-tkv9c: secret "image-registry-tls" not found Jul 08 16:03:52.963669 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963605 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32c32588-d44b-4083-8fbd-85f47cf30ea5-ca-trust-extracted\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.963669 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:52.963668 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls podName:32c32588-d44b-4083-8fbd-85f47cf30ea5 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:53.463641249 +0000 UTC m=+36.010486342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls") pod "image-registry-d67db9954-tkv9c" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5") : secret "image-registry-tls" not found Jul 08 16:03:52.963972 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.963953 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-trusted-ca\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.967265 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.967246 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-installation-pull-secrets\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.967529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.967250 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-image-registry-private-configuration\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.973682 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.973654 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-bound-sa-token\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.973780 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.973765 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2z5b\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-kube-api-access-h2z5b\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:52.993150 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.993122 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zhgft"] Jul 08 16:03:52.993235 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.993158 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:52.996038 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.996019 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 08 16:03:52.996361 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.996342 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-9gkvt\"" Jul 08 16:03:52.996438 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.996390 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 08 16:03:52.996438 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.996413 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 08 16:03:52.997408 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:52.997390 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 08 16:03:53.002668 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.002650 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 08 16:03:53.012291 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012268 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv"] Jul 08 16:03:53.012392 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012308 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777"] Jul 08 16:03:53.012392 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012328 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wp8nf"] Jul 08 16:03:53.012392 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012345 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5"] Jul 08 16:03:53.012392 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012353 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:03:53.012392 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012366 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8"] Jul 08 16:03:53.012392 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012377 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.012668 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012529 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:03:53.012742 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.012655 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:03:53.017270 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017247 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 08 16:03:53.017270 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017256 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 08 16:03:53.017408 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017285 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 08 16:03:53.017408 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017321 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 08 16:03:53.017505 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017425 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 16:03:53.017575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017556 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pr5b4\"" Jul 08 16:03:53.017639 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017581 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wzt7p\"" Jul 08 16:03:53.017691 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017674 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xz56b\"" Jul 08 16:03:53.017883 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.017789 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 16:03:53.032554 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.032538 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-qthjk"] Jul 08 16:03:53.032673 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.032659 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8" Jul 08 16:03:53.035329 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.035316 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-x9jqd\"" Jul 08 16:03:53.060620 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.060591 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx"] Jul 08 16:03:53.060749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.060632 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-vqmfn"] Jul 08 16:03:53.060749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.060654 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qthjk"] Jul 08 16:03:53.060749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.060671 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf"] Jul 08 16:03:53.060749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.060689 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8"] Jul 08 16:03:53.060749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.060708 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75f96688f5-4vp7w"] Jul 08 16:03:53.060749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.060712 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qthjk" Jul 08 16:03:53.060749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.060727 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zhgft"] Jul 08 16:03:53.063815 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.063793 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6pg7\" (UniqueName: \"kubernetes.io/projected/93739528-2d92-4533-9351-eb673aad2c08-kube-api-access-g6pg7\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:53.063957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.063827 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8r6v\" (UniqueName: \"kubernetes.io/projected/7f660b8b-1a38-4e75-b353-11842c393317-kube-api-access-m8r6v\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.063957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.063863 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhvlh\" (UniqueName: \"kubernetes.io/projected/1b73f265-f5e7-4aaa-a960-ea10101d0d55-kube-api-access-lhvlh\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.063957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.063887 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfdb2476-acb0-483c-8612-85e59e2d5303-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:53.063957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.063909 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f660b8b-1a38-4e75-b353-11842c393317-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.064222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.063965 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 08 16:03:53.064222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.063976 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0721a8f-4745-41d6-944c-d4ab9a293612-config\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:53.064222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.063997 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 08 16:03:53.064222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064006 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs9jn\" (UniqueName: \"kubernetes.io/projected/e1292322-1790-4f8c-83ed-a18cae7ef53c-kube-api-access-vs9jn\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.064222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064054 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:53.064222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064083 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 08 16:03:53.064222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064094 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zdhd9\"" Jul 08 16:03:53.064222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064107 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b73f265-f5e7-4aaa-a960-ea10101d0d55-config\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.064619 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064259 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.064619 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b73f265-f5e7-4aaa-a960-ea10101d0d55-trusted-ca\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.064619 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:53.064619 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064353 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0721a8f-4745-41d6-944c-d4ab9a293612-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:53.064619 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064614 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:53.064866 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064696 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/d0721a8f-4745-41d6-944c-d4ab9a293612-config\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:53.064866 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064792 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.065138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064875 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x524c\" (UniqueName: \"kubernetes.io/projected/474c28fe-7e30-4d21-a705-1fc25a2ee18f-kube-api-access-x524c\") pod \"volume-data-source-validator-5d456fdbcd-qslqf\" (UID: \"474c28fe-7e30-4d21-a705-1fc25a2ee18f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf" Jul 08 16:03:53.065138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064904 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:53.065138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064915 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1b73f265-f5e7-4aaa-a960-ea10101d0d55-config\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.065138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.064966 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvb8r\" (UniqueName: \"kubernetes.io/projected/d0721a8f-4745-41d6-944c-d4ab9a293612-kube-api-access-vvb8r\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:53.065138 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.065007 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 16:03:53.065138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.065022 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f660b8b-1a38-4e75-b353-11842c393317-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.065138 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.065084 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls podName:b6ea2261-c7e6-49b0-84fb-0a7a3c736f08 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:53.565052271 +0000 UTC m=+36.111897376 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8x9d5" (UID: "b6ea2261-c7e6-49b0-84fb-0a7a3c736f08") : secret "cluster-monitoring-operator-tls" not found Jul 08 16:03:53.065138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.065102 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b73f265-f5e7-4aaa-a960-ea10101d0d55-serving-cert\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.065600 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.065139 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcsjz\" (UniqueName: \"kubernetes.io/projected/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-kube-api-access-fcsjz\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:53.065600 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.065173 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-default-certificate\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.065600 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.065197 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-stats-auth\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.065600 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.065483 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:53.065802 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.065638 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1b73f265-f5e7-4aaa-a960-ea10101d0d55-trusted-ca\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.067002 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.066979 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/d0721a8f-4745-41d6-944c-d4ab9a293612-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:53.067509 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.067491 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1b73f265-f5e7-4aaa-a960-ea10101d0d55-serving-cert\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.079225 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.079206 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvb8r\" (UniqueName: \"kubernetes.io/projected/d0721a8f-4745-41d6-944c-d4ab9a293612-kube-api-access-vvb8r\") pod \"service-ca-operator-5d46d9ffd4-mv777\" (UID: \"d0721a8f-4745-41d6-944c-d4ab9a293612\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:53.079317 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.079232 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhvlh\" (UniqueName: \"kubernetes.io/projected/1b73f265-f5e7-4aaa-a960-ea10101d0d55-kube-api-access-lhvlh\") pod \"console-operator-68b757fb67-wp8nf\" (UID: \"1b73f265-f5e7-4aaa-a960-ea10101d0d55\") " pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.079873 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.079857 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcsjz\" (UniqueName: \"kubernetes.io/projected/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-kube-api-access-fcsjz\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:53.081151 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.081135 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x524c\" (UniqueName: \"kubernetes.io/projected/474c28fe-7e30-4d21-a705-1fc25a2ee18f-kube-api-access-x524c\") pod \"volume-data-source-validator-5d456fdbcd-qslqf\" (UID: \"474c28fe-7e30-4d21-a705-1fc25a2ee18f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf" Jul 08 16:03:53.101841 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.101810 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf" Jul 08 16:03:53.151208 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.151174 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:03:53.166606 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.166551 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-default-certificate\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.166606 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.166591 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-stats-auth\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.166822 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.166624 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a18c97-d257-42c6-953a-115296d85fc5-tmp-dir\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.166822 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.166652 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6pg7\" (UniqueName: \"kubernetes.io/projected/93739528-2d92-4533-9351-eb673aad2c08-kube-api-access-g6pg7\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:53.166822 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.166680 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76bba082-a1e8-4954-9011-9c6209aa5ae6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.167339 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167176 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8r6v\" (UniqueName: \"kubernetes.io/projected/7f660b8b-1a38-4e75-b353-11842c393317-kube-api-access-m8r6v\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.167339 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167235 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnwth\" (UniqueName: \"kubernetes.io/projected/49d96c6c-2207-4d00-ab5e-bf123f948a0f-kube-api-access-rnwth\") pod \"ingress-canary-qthjk\" (UID: \"49d96c6c-2207-4d00-ab5e-bf123f948a0f\") " pod="openshift-ingress-canary/ingress-canary-qthjk" Jul 08 16:03:53.167339 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167271 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76bba082-a1e8-4954-9011-9c6209aa5ae6-tmp\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.167558 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167442 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfdb2476-acb0-483c-8612-85e59e2d5303-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:53.167558 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167498 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f660b8b-1a38-4e75-b353-11842c393317-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167647 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs9jn\" (UniqueName: \"kubernetes.io/projected/e1292322-1790-4f8c-83ed-a18cae7ef53c-kube-api-access-vs9jn\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167698 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjrq6\" (UniqueName: \"kubernetes.io/projected/419195b5-2be7-4af3-a5db-a5f5ddcb5672-kube-api-access-sjrq6\") pod \"network-check-source-989f8b8dc-j86w8\" (UID: \"419195b5-2be7-4af3-a5db-a5f5ddcb5672\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167753 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54a18c97-d257-42c6-953a-115296d85fc5-config-volume\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167782 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167805 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167843 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/76bba082-a1e8-4954-9011-9c6209aa5ae6-snapshots\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167866 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j49bp\" (UniqueName: \"kubernetes.io/projected/76bba082-a1e8-4954-9011-9c6209aa5ae6-kube-api-access-j49bp\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167889 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167929 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76bba082-a1e8-4954-9011-9c6209aa5ae6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.167987 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.168005 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.168017 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdc7h\" (UniqueName: \"kubernetes.io/projected/54a18c97-d257-42c6-953a-115296d85fc5-kube-api-access-qdc7h\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.168096 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls podName:93739528-2d92-4533-9351-eb673aad2c08 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:53.668075975 +0000 UTC m=+36.214921095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-xpjfv" (UID: "93739528-2d92-4533-9351-eb673aad2c08") : secret "samples-operator-tls" not found Jul 08 16:03:53.168976 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.168119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76bba082-a1e8-4954-9011-9c6209aa5ae6-serving-cert\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.168188 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f660b8b-1a38-4e75-b353-11842c393317-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.168221 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/bfdb2476-acb0-483c-8612-85e59e2d5303-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.168243 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49d96c6c-2207-4d00-ab5e-bf123f948a0f-cert\") pod \"ingress-canary-qthjk\" (UID: \"49d96c6c-2207-4d00-ab5e-bf123f948a0f\") " pod="openshift-ingress-canary/ingress-canary-qthjk" Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.168347 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.168412 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:03:53.668394697 +0000 UTC m=+36.215239804 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : secret "router-metrics-certs-default" not found Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.168506 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.168581 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert podName:bfdb2476-acb0-483c-8612-85e59e2d5303 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:53.668568696 +0000 UTC m=+36.215413788 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-jhwzx" (UID: "bfdb2476-acb0-483c-8612-85e59e2d5303") : secret "networking-console-plugin-cert" not found Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.169292 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/7f660b8b-1a38-4e75-b353-11842c393317-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.169754 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.169720 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:03:53.669696467 +0000 UTC m=+36.216541563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : configmap references non-existent config key: service-ca.crt Jul 08 16:03:53.170206 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.170169 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/7f660b8b-1a38-4e75-b353-11842c393317-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.174791 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.174596 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" Jul 08 16:03:53.174921 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.174890 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-default-certificate\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.175296 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.175277 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-stats-auth\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.178753 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.178731 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs9jn\" (UniqueName: \"kubernetes.io/projected/e1292322-1790-4f8c-83ed-a18cae7ef53c-kube-api-access-vs9jn\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.179232 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.179188 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8r6v\" (UniqueName: \"kubernetes.io/projected/7f660b8b-1a38-4e75-b353-11842c393317-kube-api-access-m8r6v\") pod \"kube-storage-version-migrator-operator-846bf668f7-tdhsf\" (UID: \"7f660b8b-1a38-4e75-b353-11842c393317\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.179331 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.179300 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6pg7\" (UniqueName: \"kubernetes.io/projected/93739528-2d92-4533-9351-eb673aad2c08-kube-api-access-g6pg7\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:53.246862 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.246591 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" Jul 08 16:03:53.269865 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.269822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76bba082-a1e8-4954-9011-9c6209aa5ae6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.269949 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.269867 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.269949 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.269893 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdc7h\" (UniqueName: \"kubernetes.io/projected/54a18c97-d257-42c6-953a-115296d85fc5-kube-api-access-qdc7h\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.269949 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.269933 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76bba082-a1e8-4954-9011-9c6209aa5ae6-serving-cert\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.270109 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.269991 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49d96c6c-2207-4d00-ab5e-bf123f948a0f-cert\") pod \"ingress-canary-qthjk\" (UID: \"49d96c6c-2207-4d00-ab5e-bf123f948a0f\") " pod="openshift-ingress-canary/ingress-canary-qthjk" Jul 08 16:03:53.272053 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.270490 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 16:03:53.272053 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.271018 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76bba082-a1e8-4954-9011-9c6209aa5ae6-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.272714 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a18c97-d257-42c6-953a-115296d85fc5-tmp-dir\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.272776 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76bba082-a1e8-4954-9011-9c6209aa5ae6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.272814 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnwth\" (UniqueName: \"kubernetes.io/projected/49d96c6c-2207-4d00-ab5e-bf123f948a0f-kube-api-access-rnwth\") pod \"ingress-canary-qthjk\" (UID: \"49d96c6c-2207-4d00-ab5e-bf123f948a0f\") " pod="openshift-ingress-canary/ingress-canary-qthjk" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.272841 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76bba082-a1e8-4954-9011-9c6209aa5ae6-tmp\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.272915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjrq6\" (UniqueName: \"kubernetes.io/projected/419195b5-2be7-4af3-a5db-a5f5ddcb5672-kube-api-access-sjrq6\") pod \"network-check-source-989f8b8dc-j86w8\" (UID: \"419195b5-2be7-4af3-a5db-a5f5ddcb5672\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.272958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54a18c97-d257-42c6-953a-115296d85fc5-config-volume\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.273020 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/76bba082-a1e8-4954-9011-9c6209aa5ae6-snapshots\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.273051 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j49bp\" (UniqueName: \"kubernetes.io/projected/76bba082-a1e8-4954-9011-9c6209aa5ae6-kube-api-access-j49bp\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.273330 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls podName:54a18c97-d257-42c6-953a-115296d85fc5 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:53.773306564 +0000 UTC m=+36.320151664 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls") pod "dns-default-zhgft" (UID: "54a18c97-d257-42c6-953a-115296d85fc5") : secret "dns-default-metrics-tls" not found Jul 08 16:03:53.273766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.273479 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76bba082-a1e8-4954-9011-9c6209aa5ae6-tmp\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.274336 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.273845 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/76bba082-a1e8-4954-9011-9c6209aa5ae6-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.274393 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.274342 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/76bba082-a1e8-4954-9011-9c6209aa5ae6-snapshots\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.275168 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.274860 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54a18c97-d257-42c6-953a-115296d85fc5-tmp-dir\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.275168 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.275121 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/49d96c6c-2207-4d00-ab5e-bf123f948a0f-cert\") pod \"ingress-canary-qthjk\" (UID: \"49d96c6c-2207-4d00-ab5e-bf123f948a0f\") " pod="openshift-ingress-canary/ingress-canary-qthjk" Jul 08 16:03:53.275663 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.275613 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54a18c97-d257-42c6-953a-115296d85fc5-config-volume\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.279857 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.279801 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76bba082-a1e8-4954-9011-9c6209aa5ae6-serving-cert\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.298881 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.298823 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjrq6\" (UniqueName: \"kubernetes.io/projected/419195b5-2be7-4af3-a5db-a5f5ddcb5672-kube-api-access-sjrq6\") pod \"network-check-source-989f8b8dc-j86w8\" (UID: \"419195b5-2be7-4af3-a5db-a5f5ddcb5672\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8" Jul 08 16:03:53.299688 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.299630 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j49bp\" (UniqueName: \"kubernetes.io/projected/76bba082-a1e8-4954-9011-9c6209aa5ae6-kube-api-access-j49bp\") pod \"insights-operator-5c847dcfc4-vqmfn\" (UID: \"76bba082-a1e8-4954-9011-9c6209aa5ae6\") " pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.300560 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.300532 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnwth\" (UniqueName: \"kubernetes.io/projected/49d96c6c-2207-4d00-ab5e-bf123f948a0f-kube-api-access-rnwth\") pod \"ingress-canary-qthjk\" (UID: \"49d96c6c-2207-4d00-ab5e-bf123f948a0f\") " pod="openshift-ingress-canary/ingress-canary-qthjk" Jul 08 16:03:53.302594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.302331 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" Jul 08 16:03:53.303294 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.303243 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdc7h\" (UniqueName: \"kubernetes.io/projected/54a18c97-d257-42c6-953a-115296d85fc5-kube-api-access-qdc7h\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.337998 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.337652 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf"] Jul 08 16:03:53.343262 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.343044 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-wp8nf"] Jul 08 16:03:53.350017 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:53.347130 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b73f265_f5e7_4aaa_a960_ea10101d0d55.slice/crio-52d93d24f49672aa148ecf136f3b3a9a22a7438b037e8bf5a74fae45069662ea WatchSource:0}: Error finding container 52d93d24f49672aa148ecf136f3b3a9a22a7438b037e8bf5a74fae45069662ea: Status 404 returned error can't find the container with id 52d93d24f49672aa148ecf136f3b3a9a22a7438b037e8bf5a74fae45069662ea Jul 08 16:03:53.350017 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:53.348932 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod474c28fe_7e30_4d21_a705_1fc25a2ee18f.slice/crio-bd7a276c543e6d0349db1bd2fdba3b6a5117db88f1262fe3b4bb3173cb5a23fa WatchSource:0}: Error finding container bd7a276c543e6d0349db1bd2fdba3b6a5117db88f1262fe3b4bb3173cb5a23fa: Status 404 returned error can't find the container with id bd7a276c543e6d0349db1bd2fdba3b6a5117db88f1262fe3b4bb3173cb5a23fa Jul 08 16:03:53.368307 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.368283 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8" Jul 08 16:03:53.378548 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.378513 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-qthjk" Jul 08 16:03:53.383982 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.383922 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777"] Jul 08 16:03:53.392190 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.392082 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:53.394101 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:53.394050 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0721a8f_4745_41d6_944c_d4ab9a293612.slice/crio-88d3ca071a2c2401d100bad958553d16517ca8934f2ab2a726c7a424ea78af44 WatchSource:0}: Error finding container 88d3ca071a2c2401d100bad958553d16517ca8934f2ab2a726c7a424ea78af44: Status 404 returned error can't find the container with id 88d3ca071a2c2401d100bad958553d16517ca8934f2ab2a726c7a424ea78af44 Jul 08 16:03:53.444915 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.444828 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf"] Jul 08 16:03:53.448376 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:53.448323 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f660b8b_1a38_4e75_b353_11842c393317.slice/crio-7a2b2f59ef72d6f59da879141c9db0356902545b5933940ed0c0c3f9edc2a4b4 WatchSource:0}: Error finding container 7a2b2f59ef72d6f59da879141c9db0356902545b5933940ed0c0c3f9edc2a4b4: Status 404 returned error can't find the container with id 7a2b2f59ef72d6f59da879141c9db0356902545b5933940ed0c0c3f9edc2a4b4 Jul 08 16:03:53.476551 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.476437 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:53.476688 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.476598 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 16:03:53.476688 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.476621 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d67db9954-tkv9c: secret "image-registry-tls" not found Jul 08 16:03:53.476688 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.476688 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls podName:32c32588-d44b-4083-8fbd-85f47cf30ea5 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:54.47666843 +0000 UTC m=+37.023513538 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls") pod "image-registry-d67db9954-tkv9c" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5") : secret "image-registry-tls" not found Jul 08 16:03:53.498970 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.498940 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-vqmfn"] Jul 08 16:03:53.503863 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:53.503833 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76bba082_a1e8_4954_9011_9c6209aa5ae6.slice/crio-51eec58f5d60f635d7daaf26ee0e8a0d6d16c71a0471445497712d66e364e020 WatchSource:0}: Error finding container 51eec58f5d60f635d7daaf26ee0e8a0d6d16c71a0471445497712d66e364e020: Status 404 returned error can't find the container with id 51eec58f5d60f635d7daaf26ee0e8a0d6d16c71a0471445497712d66e364e020 Jul 08 16:03:53.536931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.536896 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8"] Jul 08 16:03:53.539588 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:53.539555 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod419195b5_2be7_4af3_a5db_a5f5ddcb5672.slice/crio-828c42adf1ebfc3ecec261d6510f31cc99744806025df8898034b058225e874d WatchSource:0}: Error finding container 828c42adf1ebfc3ecec261d6510f31cc99744806025df8898034b058225e874d: Status 404 returned error can't find the container with id 828c42adf1ebfc3ecec261d6510f31cc99744806025df8898034b058225e874d Jul 08 16:03:53.548546 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.548517 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-qthjk"] Jul 08 16:03:53.551225 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:03:53.551204 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49d96c6c_2207_4d00_ab5e_bf123f948a0f.slice/crio-b426cc6be6153ea779020105024707b258eecdca0fe1e133dd4af39b91614c72 WatchSource:0}: Error finding container b426cc6be6153ea779020105024707b258eecdca0fe1e133dd4af39b91614c72: Status 404 returned error can't find the container with id b426cc6be6153ea779020105024707b258eecdca0fe1e133dd4af39b91614c72 Jul 08 16:03:53.553273 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.553234 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:53.577967 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.577942 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:53.578112 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.578055 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 16:03:53.578168 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.578128 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls podName:b6ea2261-c7e6-49b0-84fb-0a7a3c736f08 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:54.578111422 +0000 UTC m=+37.124956519 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8x9d5" (UID: "b6ea2261-c7e6-49b0-84fb-0a7a3c736f08") : secret "cluster-monitoring-operator-tls" not found Jul 08 16:03:53.615696 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.615662 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:53.679240 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.679210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.679392 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.679251 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:53.679392 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.679284 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:53.679392 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.679367 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 16:03:53.679392 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.679384 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 16:03:53.679609 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.679394 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:03:54.679378117 +0000 UTC m=+37.226223213 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : configmap references non-existent config key: service-ca.crt Jul 08 16:03:53.679609 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.679448 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls podName:93739528-2d92-4533-9351-eb673aad2c08 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:54.679430035 +0000 UTC m=+37.226275161 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-xpjfv" (UID: "93739528-2d92-4533-9351-eb673aad2c08") : secret "samples-operator-tls" not found Jul 08 16:03:53.679609 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.679467 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:03:54.679457674 +0000 UTC m=+37.226302767 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : secret "router-metrics-certs-default" not found Jul 08 16:03:53.679762 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.679633 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:53.679762 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.679727 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 16:03:53.679842 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.679772 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert podName:bfdb2476-acb0-483c-8612-85e59e2d5303 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:54.679759773 +0000 UTC m=+37.226604880 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-jhwzx" (UID: "bfdb2476-acb0-483c-8612-85e59e2d5303") : secret "networking-console-plugin-cert" not found Jul 08 16:03:53.740929 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.740842 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:53.780869 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.780837 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:53.781046 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.781010 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 16:03:53.781169 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:53.781099 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls podName:54a18c97-d257-42c6-953a-115296d85fc5 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:54.781078601 +0000 UTC m=+37.327923709 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls") pod "dns-default-zhgft" (UID: "54a18c97-d257-42c6-953a-115296d85fc5") : secret "dns-default-metrics-tls" not found Jul 08 16:03:53.842275 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:53.842167 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:54.141088 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.141002 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" event={"ID":"d0721a8f-4745-41d6-944c-d4ab9a293612","Type":"ContainerStarted","Data":"88d3ca071a2c2401d100bad958553d16517ca8934f2ab2a726c7a424ea78af44"} Jul 08 16:03:54.142113 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.142087 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" event={"ID":"7f660b8b-1a38-4e75-b353-11842c393317","Type":"ContainerStarted","Data":"7a2b2f59ef72d6f59da879141c9db0356902545b5933940ed0c0c3f9edc2a4b4"} Jul 08 16:03:54.143098 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.143079 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qthjk" event={"ID":"49d96c6c-2207-4d00-ab5e-bf123f948a0f","Type":"ContainerStarted","Data":"b426cc6be6153ea779020105024707b258eecdca0fe1e133dd4af39b91614c72"} Jul 08 16:03:54.144165 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.144140 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" event={"ID":"76bba082-a1e8-4954-9011-9c6209aa5ae6","Type":"ContainerStarted","Data":"51eec58f5d60f635d7daaf26ee0e8a0d6d16c71a0471445497712d66e364e020"} Jul 08 16:03:54.145226 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.145195 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf" event={"ID":"474c28fe-7e30-4d21-a705-1fc25a2ee18f","Type":"ContainerStarted","Data":"bd7a276c543e6d0349db1bd2fdba3b6a5117db88f1262fe3b4bb3173cb5a23fa"} Jul 08 16:03:54.146271 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.146252 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" event={"ID":"1b73f265-f5e7-4aaa-a960-ea10101d0d55","Type":"ContainerStarted","Data":"52d93d24f49672aa148ecf136f3b3a9a22a7438b037e8bf5a74fae45069662ea"} Jul 08 16:03:54.147228 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.147212 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8" event={"ID":"419195b5-2be7-4af3-a5db-a5f5ddcb5672","Type":"ContainerStarted","Data":"828c42adf1ebfc3ecec261d6510f31cc99744806025df8898034b058225e874d"} Jul 08 16:03:54.488901 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.488132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:54.488901 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.488459 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 16:03:54.488901 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.488478 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d67db9954-tkv9c: secret "image-registry-tls" not found Jul 08 16:03:54.488901 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.488535 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls podName:32c32588-d44b-4083-8fbd-85f47cf30ea5 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:56.488517158 +0000 UTC m=+39.035362266 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls") pod "image-registry-d67db9954-tkv9c" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5") : secret "image-registry-tls" not found Jul 08 16:03:54.538519 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.538427 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:54.590097 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.589538 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:54.590097 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.589683 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 16:03:54.590097 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.589744 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls podName:b6ea2261-c7e6-49b0-84fb-0a7a3c736f08 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:56.589725957 +0000 UTC m=+39.136571055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8x9d5" (UID: "b6ea2261-c7e6-49b0-84fb-0a7a3c736f08") : secret "cluster-monitoring-operator-tls" not found Jul 08 16:03:54.640866 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.640807 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:54.691010 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.690237 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:54.691010 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.690293 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:54.691010 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.690327 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:54.691010 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.690367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:54.691010 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.690536 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:03:56.690517885 +0000 UTC m=+39.237362977 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : configmap references non-existent config key: service-ca.crt Jul 08 16:03:54.691010 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.690929 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 16:03:54.691010 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.690980 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert podName:bfdb2476-acb0-483c-8612-85e59e2d5303 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:56.690963174 +0000 UTC m=+39.237808270 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-jhwzx" (UID: "bfdb2476-acb0-483c-8612-85e59e2d5303") : secret "networking-console-plugin-cert" not found Jul 08 16:03:54.691516 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.691057 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 16:03:54.691516 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.691135 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls podName:93739528-2d92-4533-9351-eb673aad2c08 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:56.691115211 +0000 UTC m=+39.237960318 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-xpjfv" (UID: "93739528-2d92-4533-9351-eb673aad2c08") : secret "samples-operator-tls" not found Jul 08 16:03:54.691516 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.691057 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 16:03:54.691516 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.691177 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:03:56.691165973 +0000 UTC m=+39.238011065 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : secret "router-metrics-certs-default" not found Jul 08 16:03:54.792496 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.791720 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:54.792496 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.791914 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 16:03:54.792496 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:54.791974 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls podName:54a18c97-d257-42c6-953a-115296d85fc5 nodeName:}" failed. No retries permitted until 2026-07-08 16:03:56.791956406 +0000 UTC m=+39.338801501 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls") pod "dns-default-zhgft" (UID: "54a18c97-d257-42c6-953a-115296d85fc5") : secret "dns-default-metrics-tls" not found Jul 08 16:03:54.948519 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:54.948476 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:55.020866 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:55.020828 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:56.101552 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.101445 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:56.161153 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.161045 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf" event={"ID":"474c28fe-7e30-4d21-a705-1fc25a2ee18f","Type":"ContainerStarted","Data":"4edd85d146565d7f41c1f33a61c6603ee089c65514ce8d5bf6a39e90bfb0cb77"} Jul 08 16:03:56.183376 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.182710 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-qslqf" podStartSLOduration=27.745571893 podStartE2EDuration="30.182688825s" totalImagesPullingTime="2.437116932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:03:56.18113915 +0000 UTC m=+38.727984264" watchObservedRunningTime="2026-07-08 16:03:56.182688825 +0000 UTC m=+38.729533940" Jul 08 16:03:56.194421 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.194290 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:56.293196 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.293119 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:56.384762 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.384481 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:56.520360 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.520316 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:03:56.520524 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.520505 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 16:03:56.520524 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.520523 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d67db9954-tkv9c: secret "image-registry-tls" not found Jul 08 16:03:56.520623 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.520581 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls podName:32c32588-d44b-4083-8fbd-85f47cf30ea5 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:00.520560641 +0000 UTC m=+43.067405754 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls") pod "image-registry-d67db9954-tkv9c" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5") : secret "image-registry-tls" not found Jul 08 16:03:56.621147 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.621046 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:03:56.621308 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.621177 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 16:03:56.621308 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.621245 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls podName:b6ea2261-c7e6-49b0-84fb-0a7a3c736f08 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:00.62122711 +0000 UTC m=+43.168072219 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8x9d5" (UID: "b6ea2261-c7e6-49b0-84fb-0a7a3c736f08") : secret "cluster-monitoring-operator-tls" not found Jul 08 16:03:56.722474 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.722441 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:03:56.722653 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.722493 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:56.722653 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.722522 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:03:56.722653 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.722555 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:03:56.722653 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.722623 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 16:03:56.722847 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.722678 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 16:03:56.722847 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.722699 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert podName:bfdb2476-acb0-483c-8612-85e59e2d5303 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:00.722678297 +0000 UTC m=+43.269523407 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-jhwzx" (UID: "bfdb2476-acb0-483c-8612-85e59e2d5303") : secret "networking-console-plugin-cert" not found Jul 08 16:03:56.722847 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.722715 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls podName:93739528-2d92-4533-9351-eb673aad2c08 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:00.722705861 +0000 UTC m=+43.269550952 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-xpjfv" (UID: "93739528-2d92-4533-9351-eb673aad2c08") : secret "samples-operator-tls" not found Jul 08 16:03:56.722847 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.722624 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 16:03:56.722847 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.722749 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:04:00.722740077 +0000 UTC m=+43.269585169 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : configmap references non-existent config key: service-ca.crt Jul 08 16:03:56.722847 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.722776 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:04:00.722758641 +0000 UTC m=+43.269603752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : secret "router-metrics-certs-default" not found Jul 08 16:03:56.823567 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:56.823440 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:03:56.823742 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.823605 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 16:03:56.823742 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:03:56.823677 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls podName:54a18c97-d257-42c6-953a-115296d85fc5 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:00.823656447 +0000 UTC m=+43.370501542 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls") pod "dns-default-zhgft" (UID: "54a18c97-d257-42c6-953a-115296d85fc5") : secret "dns-default-metrics-tls" not found Jul 08 16:03:57.143850 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:57.143773 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:57.243374 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:57.243343 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:58.012850 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:58.012811 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:03:58.081192 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:03:58.081158 2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" limit=1024 Jul 08 16:04:00.556366 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:00.556326 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:04:00.556842 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.556481 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 16:04:00.556842 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.556506 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d67db9954-tkv9c: secret "image-registry-tls" not found Jul 08 16:04:00.556842 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.556568 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls podName:32c32588-d44b-4083-8fbd-85f47cf30ea5 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:08.556548952 +0000 UTC m=+51.103394051 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls") pod "image-registry-d67db9954-tkv9c" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5") : secret "image-registry-tls" not found Jul 08 16:04:00.657387 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:00.657350 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:04:00.657544 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.657502 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 16:04:00.657580 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.657567 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls podName:b6ea2261-c7e6-49b0-84fb-0a7a3c736f08 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:08.657551976 +0000 UTC m=+51.204397069 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8x9d5" (UID: "b6ea2261-c7e6-49b0-84fb-0a7a3c736f08") : secret "cluster-monitoring-operator-tls" not found Jul 08 16:04:00.758686 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:00.758650 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:04:00.758686 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:00.758688 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:00.758868 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:00.758711 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:04:00.758868 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:00.758732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:00.758868 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.758809 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 16:04:00.758868 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.758831 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 16:04:00.758868 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.758848 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:04:08.758835999 +0000 UTC m=+51.305681090 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : configmap references non-existent config key: service-ca.crt Jul 08 16:04:00.758868 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.758870 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:04:08.758857099 +0000 UTC m=+51.305702190 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : secret "router-metrics-certs-default" not found Jul 08 16:04:00.759117 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.758813 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 16:04:00.759117 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.758883 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls podName:93739528-2d92-4533-9351-eb673aad2c08 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:08.758877593 +0000 UTC m=+51.305722685 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-xpjfv" (UID: "93739528-2d92-4533-9351-eb673aad2c08") : secret "samples-operator-tls" not found Jul 08 16:04:00.759117 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.758903 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert podName:bfdb2476-acb0-483c-8612-85e59e2d5303 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:08.758891796 +0000 UTC m=+51.305736887 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-jhwzx" (UID: "bfdb2476-acb0-483c-8612-85e59e2d5303") : secret "networking-console-plugin-cert" not found Jul 08 16:04:00.859644 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:00.859571 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:04:00.859780 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.859722 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 16:04:00.859818 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:00.859781 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls podName:54a18c97-d257-42c6-953a-115296d85fc5 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:08.859763977 +0000 UTC m=+51.406609077 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls") pod "dns-default-zhgft" (UID: "54a18c97-d257-42c6-953a-115296d85fc5") : secret "dns-default-metrics-tls" not found Jul 08 16:04:04.183938 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.183903 2563 generic.go:356] "Generic (PLEG): container finished" podID="ad541a88-86e4-4477-9db5-d361b00f1268" containerID="05fff0963984d5c9d932efe4d258b8a7de80542eae540049e866daeb4c52210d" exitCode=0 Jul 08 16:04:04.184464 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.183980 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerDied","Data":"05fff0963984d5c9d932efe4d258b8a7de80542eae540049e866daeb4c52210d"} Jul 08 16:04:04.186282 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.186253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-qthjk" event={"ID":"49d96c6c-2207-4d00-ab5e-bf123f948a0f","Type":"ContainerStarted","Data":"b06fb9e4cf8ecb545adcf3e684b27dcce9cbe4a4e4a52a6897ec7ef12e99162d"} Jul 08 16:04:04.187857 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.187833 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" event={"ID":"76bba082-a1e8-4954-9011-9c6209aa5ae6","Type":"ContainerStarted","Data":"50ce60d41038fb78e33d58c18b4658a26ee889f93b573efb228c4ef9ae12164c"} Jul 08 16:04:04.190132 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.190114 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/0.log" Jul 08 16:04:04.190233 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.190150 2563 generic.go:356] "Generic (PLEG): container finished" podID="1b73f265-f5e7-4aaa-a960-ea10101d0d55" containerID="f367669ed94c165d14200383faad3ee3dc0a288304a3debd0178d01b424bed0b" exitCode=255 Jul 08 16:04:04.190363 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.190340 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" event={"ID":"1b73f265-f5e7-4aaa-a960-ea10101d0d55","Type":"ContainerDied","Data":"f367669ed94c165d14200383faad3ee3dc0a288304a3debd0178d01b424bed0b"} Jul 08 16:04:04.190454 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.190439 2563 scope.go:117] "RemoveContainer" containerID="f367669ed94c165d14200383faad3ee3dc0a288304a3debd0178d01b424bed0b" Jul 08 16:04:04.191984 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.191952 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8" event={"ID":"419195b5-2be7-4af3-a5db-a5f5ddcb5672","Type":"ContainerStarted","Data":"0ed78026fb2808aebaacf4b410f62c9b745bfc64f233182fe1563d1d4f341fa6"} Jul 08 16:04:04.193846 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.193824 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" event={"ID":"d0721a8f-4745-41d6-944c-d4ab9a293612","Type":"ContainerStarted","Data":"4d253a5e88816b75b565e6e05e89ef66ceef1ca7f474c6fc7a4fa6368074309f"} Jul 08 16:04:04.195366 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.195347 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" event={"ID":"7f660b8b-1a38-4e75-b353-11842c393317","Type":"ContainerStarted","Data":"d1e1a27b5d877ae39c9dab2926b85f79212b3be6d4dd513e4fb36aeb8cb2b26e"} Jul 08 16:04:04.258768 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.258718 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-j86w8" podStartSLOduration=27.409424679 podStartE2EDuration="37.258697869s" totalImagesPullingTime="9.84927319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:04.233235429 +0000 UTC m=+46.780080544" watchObservedRunningTime="2026-07-08 16:04:04.258697869 +0000 UTC m=+46.805542984" Jul 08 16:04:04.279323 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.279275 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" podStartSLOduration=27.283590325 podStartE2EDuration="37.279257956s" totalImagesPullingTime="9.995667631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:04.278424613 +0000 UTC m=+46.825269731" watchObservedRunningTime="2026-07-08 16:04:04.279257956 +0000 UTC m=+46.826103047" Jul 08 16:04:04.298985 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.298938 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-qthjk" podStartSLOduration=2.462852399 podStartE2EDuration="12.29891584s" totalImagesPullingTime="9.836063441s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:04.297319389 +0000 UTC m=+46.844164504" watchObservedRunningTime="2026-07-08 16:04:04.29891584 +0000 UTC m=+46.845760958" Jul 08 16:04:04.319442 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.319386 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" podStartSLOduration=27.434608441 podStartE2EDuration="37.319364257s" totalImagesPullingTime="9.884755816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:04.319115459 +0000 UTC m=+46.865960578" watchObservedRunningTime="2026-07-08 16:04:04.319364257 +0000 UTC m=+46.866209372" Jul 08 16:04:04.344420 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:04.343769 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" podStartSLOduration=27.406933503 podStartE2EDuration="37.343746589s" totalImagesPullingTime="9.936813086s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:04.343680046 +0000 UTC m=+46.890525162" watchObservedRunningTime="2026-07-08 16:04:04.343746589 +0000 UTC m=+46.890591705" Jul 08 16:04:05.136732 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.136695 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7"] Jul 08 16:04:05.161501 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.161467 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7"] Jul 08 16:04:05.161650 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.161601 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" Jul 08 16:04:05.164986 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.164942 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" Jul 08 16:04:05.164986 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.164975 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-ln2ql\"" Jul 08 16:04:05.165831 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.165814 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" Jul 08 16:04:05.196624 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.196593 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdshp\" (UniqueName: \"kubernetes.io/projected/0923cb15-53d6-4f05-a743-efbca787729d-kube-api-access-qdshp\") pod \"migrator-d57d58cc9-xd8z7\" (UID: \"0923cb15-53d6-4f05-a743-efbca787729d\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" Jul 08 16:04:05.203403 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.203374 2563 generic.go:356] "Generic (PLEG): container finished" podID="ad541a88-86e4-4477-9db5-d361b00f1268" containerID="60a94b6f7a9ead25102c488899269c408a7903a09c94635ac22a33e9ed012412" exitCode=0 Jul 08 16:04:05.203541 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.203450 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerDied","Data":"60a94b6f7a9ead25102c488899269c408a7903a09c94635ac22a33e9ed012412"} Jul 08 16:04:05.205203 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.205186 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:04:05.205654 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.205637 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/0.log" Jul 08 16:04:05.205724 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.205676 2563 generic.go:356] "Generic (PLEG): container finished" podID="1b73f265-f5e7-4aaa-a960-ea10101d0d55" containerID="03ac5e2497f793aad76d4f5b304f5575b57ae802e25c78e450f666b2828e46e4" exitCode=255 Jul 08 16:04:05.205781 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.205732 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" event={"ID":"1b73f265-f5e7-4aaa-a960-ea10101d0d55","Type":"ContainerDied","Data":"03ac5e2497f793aad76d4f5b304f5575b57ae802e25c78e450f666b2828e46e4"} Jul 08 16:04:05.205781 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.205770 2563 scope.go:117] "RemoveContainer" containerID="f367669ed94c165d14200383faad3ee3dc0a288304a3debd0178d01b424bed0b" Jul 08 16:04:05.206001 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.205984 2563 scope.go:117] "RemoveContainer" containerID="03ac5e2497f793aad76d4f5b304f5575b57ae802e25c78e450f666b2828e46e4" Jul 08 16:04:05.206249 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:05.206200 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wp8nf_openshift-console-operator(1b73f265-f5e7-4aaa-a960-ea10101d0d55)\"" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" podUID="1b73f265-f5e7-4aaa-a960-ea10101d0d55" Jul 08 16:04:05.298762 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.298729 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdshp\" (UniqueName: \"kubernetes.io/projected/0923cb15-53d6-4f05-a743-efbca787729d-kube-api-access-qdshp\") pod \"migrator-d57d58cc9-xd8z7\" (UID: \"0923cb15-53d6-4f05-a743-efbca787729d\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" Jul 08 16:04:05.309339 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.309310 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdshp\" (UniqueName: \"kubernetes.io/projected/0923cb15-53d6-4f05-a743-efbca787729d-kube-api-access-qdshp\") pod \"migrator-d57d58cc9-xd8z7\" (UID: \"0923cb15-53d6-4f05-a743-efbca787729d\") " pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" Jul 08 16:04:05.473311 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.473272 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" Jul 08 16:04:05.626860 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.626811 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7"] Jul 08 16:04:05.629623 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:05.629592 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0923cb15_53d6_4f05_a743_efbca787729d.slice/crio-e7e70382670718585c4c172ccca0914907211c7fee23fdad81e8b3e3117c9559 WatchSource:0}: Error finding container e7e70382670718585c4c172ccca0914907211c7fee23fdad81e8b3e3117c9559: Status 404 returned error can't find the container with id e7e70382670718585c4c172ccca0914907211c7fee23fdad81e8b3e3117c9559 Jul 08 16:04:05.734155 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.732946 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vmrh8"] Jul 08 16:04:05.764878 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.764844 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmrh8"] Jul 08 16:04:05.765008 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.764972 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.767463 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.767434 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 08 16:04:05.767848 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.767832 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 08 16:04:05.768265 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.768245 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2pd8l\"" Jul 08 16:04:05.805981 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.805608 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdngw\" (UniqueName: \"kubernetes.io/projected/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-api-access-cdngw\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.805981 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.805702 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-data-volume\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.805981 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.805760 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.805981 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.805793 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-crio-socket\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.805981 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.805850 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.907178 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.907143 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdngw\" (UniqueName: \"kubernetes.io/projected/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-api-access-cdngw\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.907355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.907208 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-data-volume\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.907355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.907246 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.907355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.907264 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-crio-socket\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.907355 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.907288 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.907504 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:05.907366 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 16:04:05.907504 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:05.907407 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 16:04:05.907504 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:05.907443 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm podName:65d04972-2959-4b2b-b6b7-5f2ba538cc1d nodeName:}" failed. No retries permitted until 2026-07-08 16:04:06.407422839 +0000 UTC m=+48.954267947 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmrh8" (UID: "65d04972-2959-4b2b-b6b7-5f2ba538cc1d") : configmap "kube-rbac-proxy" not found Jul 08 16:04:05.907504 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:05.907464 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls podName:65d04972-2959-4b2b-b6b7-5f2ba538cc1d nodeName:}" failed. No retries permitted until 2026-07-08 16:04:06.407454333 +0000 UTC m=+48.954299430 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmrh8" (UID: "65d04972-2959-4b2b-b6b7-5f2ba538cc1d") : secret "insights-runtime-extractor-tls" not found Jul 08 16:04:05.907504 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.907497 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-crio-socket\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.907656 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.907641 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-data-volume\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:05.917836 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:05.917804 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdngw\" (UniqueName: \"kubernetes.io/projected/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-api-access-cdngw\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:06.166078 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.166036 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz"] Jul 08 16:04:06.181559 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.181534 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.187460 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.187436 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\"" Jul 08 16:04:06.188201 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.188182 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" Jul 08 16:04:06.188308 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.188244 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" Jul 08 16:04:06.188533 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.188518 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" Jul 08 16:04:06.193267 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.193248 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz"] Jul 08 16:04:06.207383 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.207357 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6"] Jul 08 16:04:06.213840 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.213820 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:04:06.214181 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.214154 2563 scope.go:117] "RemoveContainer" containerID="03ac5e2497f793aad76d4f5b304f5575b57ae802e25c78e450f666b2828e46e4" Jul 08 16:04:06.214311 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:06.214294 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wp8nf_openshift-console-operator(1b73f265-f5e7-4aaa-a960-ea10101d0d55)\"" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" podUID="1b73f265-f5e7-4aaa-a960-ea10101d0d55" Jul 08 16:04:06.229710 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.229681 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" event={"ID":"ad541a88-86e4-4477-9db5-d361b00f1268","Type":"ContainerStarted","Data":"52c24b1b0d48ec7316178a41b834b36c2e77c41937feec77fa1606fd32c37044"} Jul 08 16:04:06.229822 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.229722 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw"] Jul 08 16:04:06.229822 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.229790 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.232829 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.232809 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" Jul 08 16:04:06.232932 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.232834 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" Jul 08 16:04:06.233045 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.233031 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" Jul 08 16:04:06.233127 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.233094 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" Jul 08 16:04:06.245884 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.245861 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" event={"ID":"0923cb15-53d6-4f05-a743-efbca787729d","Type":"ContainerStarted","Data":"e7e70382670718585c4c172ccca0914907211c7fee23fdad81e8b3e3117c9559"} Jul 08 16:04:06.245958 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.245873 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" Jul 08 16:04:06.254029 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.254010 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\"" Jul 08 16:04:06.258638 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.258618 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-qv245\"" Jul 08 16:04:06.259004 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.258983 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6"] Jul 08 16:04:06.260093 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.260060 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw"] Jul 08 16:04:06.289446 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.286937 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ztg8s" podStartSLOduration=9.405017811 podStartE2EDuration="48.286918669s" totalImagesPullingTime="38.881900858s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:06.28452437 +0000 UTC m=+48.831369480" watchObservedRunningTime="2026-07-08 16:04:06.286918669 +0000 UTC m=+48.833763783" Jul 08 16:04:06.310377 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.310348 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-hub\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.310525 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.310462 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-additional-service-ca\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.310603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.310536 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-tmp\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.310603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.310561 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-ca\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.310767 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.310742 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.310932 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.310914 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.311102 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.311080 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fp5cs\" (UniqueName: \"kubernetes.io/projected/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-kube-api-access-fp5cs\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.311192 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.311162 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.311235 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.311193 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/093ce9f9-f671-4a90-814b-cb5f42e6ad37-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw\" (UID: \"093ce9f9-f671-4a90-814b-cb5f42e6ad37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" Jul 08 16:04:06.311274 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.311232 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdlqk\" (UniqueName: \"kubernetes.io/projected/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-kube-api-access-gdlqk\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.311274 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.311263 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2r7w\" (UniqueName: \"kubernetes.io/projected/093ce9f9-f671-4a90-814b-cb5f42e6ad37-kube-api-access-t2r7w\") pod \"managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw\" (UID: \"093ce9f9-f671-4a90-814b-cb5f42e6ad37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" Jul 08 16:04:06.412555 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.412520 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-additional-service-ca\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.412750 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.412585 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-tmp\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.412750 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.412608 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-ca\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.412750 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.412655 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:06.412750 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.412699 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:06.412973 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.412748 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.412973 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:06.412833 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 16:04:06.412973 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:06.412881 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 16:04:06.412973 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:06.412900 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls podName:65d04972-2959-4b2b-b6b7-5f2ba538cc1d nodeName:}" failed. No retries permitted until 2026-07-08 16:04:07.412885171 +0000 UTC m=+49.959730263 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmrh8" (UID: "65d04972-2959-4b2b-b6b7-5f2ba538cc1d") : secret "insights-runtime-extractor-tls" not found Jul 08 16:04:06.412973 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.412939 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.412973 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:06.412970 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm podName:65d04972-2959-4b2b-b6b7-5f2ba538cc1d nodeName:}" failed. No retries permitted until 2026-07-08 16:04:07.41295164 +0000 UTC m=+49.959796747 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmrh8" (UID: "65d04972-2959-4b2b-b6b7-5f2ba538cc1d") : configmap "kube-rbac-proxy" not found Jul 08 16:04:06.413323 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.413007 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-tmp\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.413323 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.413034 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fp5cs\" (UniqueName: \"kubernetes.io/projected/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-kube-api-access-fp5cs\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.413323 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.413112 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.413323 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.413146 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/093ce9f9-f671-4a90-814b-cb5f42e6ad37-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw\" (UID: \"093ce9f9-f671-4a90-814b-cb5f42e6ad37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" Jul 08 16:04:06.413323 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.413173 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdlqk\" (UniqueName: \"kubernetes.io/projected/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-kube-api-access-gdlqk\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.413323 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.413201 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2r7w\" (UniqueName: \"kubernetes.io/projected/093ce9f9-f671-4a90-814b-cb5f42e6ad37-kube-api-access-t2r7w\") pod \"managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw\" (UID: \"093ce9f9-f671-4a90-814b-cb5f42e6ad37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" Jul 08 16:04:06.413323 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.413249 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-hub\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.413661 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.413378 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-additional-service-ca\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.415432 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.415407 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.415685 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.415663 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-ca\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.415883 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.415854 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.415964 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.415909 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/093ce9f9-f671-4a90-814b-cb5f42e6ad37-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw\" (UID: \"093ce9f9-f671-4a90-814b-cb5f42e6ad37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" Jul 08 16:04:06.416006 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.415977 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-hub\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.416261 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.416224 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-klusterlet-config\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.424513 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.424487 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2r7w\" (UniqueName: \"kubernetes.io/projected/093ce9f9-f671-4a90-814b-cb5f42e6ad37-kube-api-access-t2r7w\") pod \"managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw\" (UID: \"093ce9f9-f671-4a90-814b-cb5f42e6ad37\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" Jul 08 16:04:06.424890 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.424871 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fp5cs\" (UniqueName: \"kubernetes.io/projected/c9fcff4a-b92c-4883-ae62-6a74b4f1e89d-kube-api-access-fp5cs\") pod \"klusterlet-addon-workmgr-5cc85bb656-fldgz\" (UID: \"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.427545 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.427530 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdlqk\" (UniqueName: \"kubernetes.io/projected/609d11f4-ffaa-4b25-a75b-51ede4fd7f03-kube-api-access-gdlqk\") pod \"cluster-proxy-proxy-agent-88968469f-sfck6\" (UID: \"609d11f4-ffaa-4b25-a75b-51ede4fd7f03\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.490541 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.490512 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:06.552647 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.552616 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:06.559682 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.559650 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" Jul 08 16:04:06.661253 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.661231 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz"] Jul 08 16:04:06.663937 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:06.663903 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9fcff4a_b92c_4883_ae62_6a74b4f1e89d.slice/crio-32374c44ab7118041e27776ecc02757dade159e777f2f785818a0d7f2fcb7cbe WatchSource:0}: Error finding container 32374c44ab7118041e27776ecc02757dade159e777f2f785818a0d7f2fcb7cbe: Status 404 returned error can't find the container with id 32374c44ab7118041e27776ecc02757dade159e777f2f785818a0d7f2fcb7cbe Jul 08 16:04:06.725084 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.725043 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6"] Jul 08 16:04:06.728190 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:06.728161 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod609d11f4_ffaa_4b25_a75b_51ede4fd7f03.slice/crio-e8c533d93802f25a0546d09dc4659c2a23986ca4c0db29cde37cbfe4b8757323 WatchSource:0}: Error finding container e8c533d93802f25a0546d09dc4659c2a23986ca4c0db29cde37cbfe4b8757323: Status 404 returned error can't find the container with id e8c533d93802f25a0546d09dc4659c2a23986ca4c0db29cde37cbfe4b8757323 Jul 08 16:04:06.755329 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:06.755308 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw"] Jul 08 16:04:06.757011 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:06.756986 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod093ce9f9_f671_4a90_814b_cb5f42e6ad37.slice/crio-6c4eeda75380990513d33c119ccf4fc41859e47e318afd6fd719a21bbbaf6e14 WatchSource:0}: Error finding container 6c4eeda75380990513d33c119ccf4fc41859e47e318afd6fd719a21bbbaf6e14: Status 404 returned error can't find the container with id 6c4eeda75380990513d33c119ccf4fc41859e47e318afd6fd719a21bbbaf6e14 Jul 08 16:04:07.220688 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:07.220648 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" event={"ID":"093ce9f9-f671-4a90-814b-cb5f42e6ad37","Type":"ContainerStarted","Data":"6c4eeda75380990513d33c119ccf4fc41859e47e318afd6fd719a21bbbaf6e14"} Jul 08 16:04:07.222018 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:07.221992 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" event={"ID":"609d11f4-ffaa-4b25-a75b-51ede4fd7f03","Type":"ContainerStarted","Data":"e8c533d93802f25a0546d09dc4659c2a23986ca4c0db29cde37cbfe4b8757323"} Jul 08 16:04:07.223368 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:07.223345 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" event={"ID":"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d","Type":"ContainerStarted","Data":"32374c44ab7118041e27776ecc02757dade159e777f2f785818a0d7f2fcb7cbe"} Jul 08 16:04:07.422483 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:07.422444 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:07.422658 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:07.422502 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:07.422658 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:07.422593 2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 08 16:04:07.422658 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:07.422651 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 16:04:07.422789 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:07.422672 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm podName:65d04972-2959-4b2b-b6b7-5f2ba538cc1d nodeName:}" failed. No retries permitted until 2026-07-08 16:04:09.422650686 +0000 UTC m=+51.969495791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm") pod "insights-runtime-extractor-vmrh8" (UID: "65d04972-2959-4b2b-b6b7-5f2ba538cc1d") : configmap "kube-rbac-proxy" not found Jul 08 16:04:07.422789 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:07.422700 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls podName:65d04972-2959-4b2b-b6b7-5f2ba538cc1d nodeName:}" failed. No retries permitted until 2026-07-08 16:04:09.422684087 +0000 UTC m=+51.969529178 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmrh8" (UID: "65d04972-2959-4b2b-b6b7-5f2ba538cc1d") : secret "insights-runtime-extractor-tls" not found Jul 08 16:04:07.923055 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:07.922926 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jnp8q_e6057259-93e1-4083-b7c0-863ea9fff32c/dns-node-resolver/0.log" Jul 08 16:04:08.179452 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.178135 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-758f599c5-z5xsn"] Jul 08 16:04:08.182139 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.181354 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.184458 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.184429 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-jkcv6\"" Jul 08 16:04:08.184647 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.184631 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\"" Jul 08 16:04:08.185458 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.185437 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\"" Jul 08 16:04:08.185562 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.185483 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\"" Jul 08 16:04:08.185562 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.185515 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\"" Jul 08 16:04:08.192397 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.191955 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-z5xsn"] Jul 08 16:04:08.238371 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.238309 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2204690-cd84-4abe-ad80-e631905b0738-signing-key\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.239192 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.239154 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2204690-cd84-4abe-ad80-e631905b0738-signing-cabundle\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.239299 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.239222 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8gjtm\" (UniqueName: \"kubernetes.io/projected/c2204690-cd84-4abe-ad80-e631905b0738-kube-api-access-8gjtm\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.341378 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.340863 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2204690-cd84-4abe-ad80-e631905b0738-signing-key\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.341378 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.341045 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2204690-cd84-4abe-ad80-e631905b0738-signing-cabundle\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.341378 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.341092 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8gjtm\" (UniqueName: \"kubernetes.io/projected/c2204690-cd84-4abe-ad80-e631905b0738-kube-api-access-8gjtm\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.342295 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.342217 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c2204690-cd84-4abe-ad80-e631905b0738-signing-cabundle\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.346138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.346090 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c2204690-cd84-4abe-ad80-e631905b0738-signing-key\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.350410 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.350366 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8gjtm\" (UniqueName: \"kubernetes.io/projected/c2204690-cd84-4abe-ad80-e631905b0738-kube-api-access-8gjtm\") pod \"service-ca-758f599c5-z5xsn\" (UID: \"c2204690-cd84-4abe-ad80-e631905b0738\") " pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.501294 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.501241 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-758f599c5-z5xsn" Jul 08 16:04:08.524092 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.524055 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-27pnz_849b9b0e-e00b-4793-aba1-38110bc88117/node-ca/0.log" Jul 08 16:04:08.644738 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.644706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:04:08.645037 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.645017 2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 08 16:04:08.645140 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.645043 2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-d67db9954-tkv9c: secret "image-registry-tls" not found Jul 08 16:04:08.645195 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.645158 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls podName:32c32588-d44b-4083-8fbd-85f47cf30ea5 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:24.645136678 +0000 UTC m=+67.191981793 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls") pod "image-registry-d67db9954-tkv9c" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5") : secret "image-registry-tls" not found Jul 08 16:04:08.718363 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.716399 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-758f599c5-z5xsn"] Jul 08 16:04:08.725690 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:08.724128 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2204690_cd84_4abe_ad80_e631905b0738.slice/crio-6af26a936746cd467b9791714fef623d9d8f482fd201941af4dc085b6ee7b97c WatchSource:0}: Error finding container 6af26a936746cd467b9791714fef623d9d8f482fd201941af4dc085b6ee7b97c: Status 404 returned error can't find the container with id 6af26a936746cd467b9791714fef623d9d8f482fd201941af4dc085b6ee7b97c Jul 08 16:04:08.745786 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.745548 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:04:08.745786 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.745689 2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 08 16:04:08.745786 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.745746 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls podName:b6ea2261-c7e6-49b0-84fb-0a7a3c736f08 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:24.74572725 +0000 UTC m=+67.292572366 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-8x9d5" (UID: "b6ea2261-c7e6-49b0-84fb-0a7a3c736f08") : secret "cluster-monitoring-operator-tls" not found Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.847144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.847207 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.847371 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.847403 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.847516 2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.847566 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:04:24.847549103 +0000 UTC m=+67.394394211 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : secret "router-metrics-certs-default" not found Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.847922 2563 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.847971 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls podName:93739528-2d92-4533-9351-eb673aad2c08 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:24.847955056 +0000 UTC m=+67.394800165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-xpjfv" (UID: "93739528-2d92-4533-9351-eb673aad2c08") : secret "samples-operator-tls" not found Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.848054 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle podName:e1292322-1790-4f8c-83ed-a18cae7ef53c nodeName:}" failed. No retries permitted until 2026-07-08 16:04:24.848044085 +0000 UTC m=+67.394889194 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle") pod "router-default-75f96688f5-4vp7w" (UID: "e1292322-1790-4f8c-83ed-a18cae7ef53c") : configmap references non-existent config key: service-ca.crt Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.848130 2563 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 08 16:04:08.848190 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.848164 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert podName:bfdb2476-acb0-483c-8612-85e59e2d5303 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:24.848153965 +0000 UTC m=+67.394999060 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-jhwzx" (UID: "bfdb2476-acb0-483c-8612-85e59e2d5303") : secret "networking-console-plugin-cert" not found Jul 08 16:04:08.948694 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:08.948629 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:04:08.949411 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.948870 2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 08 16:04:08.949411 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:08.948940 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls podName:54a18c97-d257-42c6-953a-115296d85fc5 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:24.948919727 +0000 UTC m=+67.495764821 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls") pod "dns-default-zhgft" (UID: "54a18c97-d257-42c6-953a-115296d85fc5") : secret "dns-default-metrics-tls" not found Jul 08 16:04:09.247118 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.246517 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-z5xsn" event={"ID":"c2204690-cd84-4abe-ad80-e631905b0738","Type":"ContainerStarted","Data":"0f547c20de721b56e21e150d977574fe4e2d368df47914be52b57f66643620bd"} Jul 08 16:04:09.247118 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.246577 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-758f599c5-z5xsn" event={"ID":"c2204690-cd84-4abe-ad80-e631905b0738","Type":"ContainerStarted","Data":"6af26a936746cd467b9791714fef623d9d8f482fd201941af4dc085b6ee7b97c"} Jul 08 16:04:09.250291 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.249806 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" event={"ID":"0923cb15-53d6-4f05-a743-efbca787729d","Type":"ContainerStarted","Data":"cc8ad9737f916edcd68c68872fed30c6ca95219e9d265d3c75e258e5e86bb00b"} Jul 08 16:04:09.250291 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.249859 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" event={"ID":"0923cb15-53d6-4f05-a743-efbca787729d","Type":"ContainerStarted","Data":"0057420aa8b319dbec85171b8e74fcf2b4c9dcda9cd40a509310c9de6249ffe4"} Jul 08 16:04:09.276152 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.275326 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-758f599c5-z5xsn" podStartSLOduration=1.2753055629999999 podStartE2EDuration="1.275305563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:09.273595156 +0000 UTC m=+51.820440271" watchObservedRunningTime="2026-07-08 16:04:09.275305563 +0000 UTC m=+51.822150679" Jul 08 16:04:09.302735 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.301965 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-d57d58cc9-xd8z7" podStartSLOduration=1.6031700660000001 podStartE2EDuration="4.301945598s" totalImagesPullingTime="2.698775532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:04:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:09.301931801 +0000 UTC m=+51.848776915" watchObservedRunningTime="2026-07-08 16:04:09.301945598 +0000 UTC m=+51.848790712" Jul 08 16:04:09.326042 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.325997 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qthjk_49d96c6c-2207-4d00-ab5e-bf123f948a0f/serve-healthcheck-canary/0.log" Jul 08 16:04:09.456401 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.455053 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:09.456401 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.455314 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:09.456401 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.455868 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:09.456401 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:09.455979 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 16:04:09.456401 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:09.456038 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls podName:65d04972-2959-4b2b-b6b7-5f2ba538cc1d nodeName:}" failed. No retries permitted until 2026-07-08 16:04:13.456017912 +0000 UTC m=+56.002863018 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmrh8" (UID: "65d04972-2959-4b2b-b6b7-5f2ba538cc1d") : secret "insights-runtime-extractor-tls" not found Jul 08 16:04:09.524651 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.524561 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-xd8z7_0923cb15-53d6-4f05-a743-efbca787729d/migrator/0.log" Jul 08 16:04:09.723826 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.723767 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-xd8z7_0923cb15-53d6-4f05-a743-efbca787729d/graceful-termination/0.log" Jul 08 16:04:09.926319 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:09.926195 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-tdhsf_7f660b8b-1a38-4e75-b353-11842c393317/kube-storage-version-migrator-operator/0.log" Jul 08 16:04:13.152304 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:13.152261 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:04:13.152304 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:13.152317 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:04:13.152874 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:13.152740 2563 scope.go:117] "RemoveContainer" containerID="03ac5e2497f793aad76d4f5b304f5575b57ae802e25c78e450f666b2828e46e4" Jul 08 16:04:13.152964 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:13.152943 2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-wp8nf_openshift-console-operator(1b73f265-f5e7-4aaa-a960-ea10101d0d55)\"" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" podUID="1b73f265-f5e7-4aaa-a960-ea10101d0d55" Jul 08 16:04:13.504179 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:13.503585 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:13.504179 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:13.503923 2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 08 16:04:13.504179 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:13.504003 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls podName:65d04972-2959-4b2b-b6b7-5f2ba538cc1d nodeName:}" failed. No retries permitted until 2026-07-08 16:04:21.503981343 +0000 UTC m=+64.050826436 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls") pod "insights-runtime-extractor-vmrh8" (UID: "65d04972-2959-4b2b-b6b7-5f2ba538cc1d") : secret "insights-runtime-extractor-tls" not found Jul 08 16:04:14.274109 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.274012 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" event={"ID":"093ce9f9-f671-4a90-814b-cb5f42e6ad37","Type":"ContainerStarted","Data":"3b9c9509cce1fc1701156c7c00ac484d74ddacb824e5c7f8458078e0cfd90301"} Jul 08 16:04:14.276086 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.276042 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" event={"ID":"609d11f4-ffaa-4b25-a75b-51ede4fd7f03","Type":"ContainerStarted","Data":"b0e68eb6ddd6c83731b71bd4a418096ec0dcc984870a4e250327b1e6722434eb"} Jul 08 16:04:14.276214 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.276090 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" event={"ID":"609d11f4-ffaa-4b25-a75b-51ede4fd7f03","Type":"ContainerStarted","Data":"82a4b314faece0327a3f5f079c2b247c5d15919078b5ac433d4453a55a420160"} Jul 08 16:04:14.276214 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.276103 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" event={"ID":"609d11f4-ffaa-4b25-a75b-51ede4fd7f03","Type":"ContainerStarted","Data":"98a868b8b14723bdc265dec3400818c749a9d286b41b5b6c2127da2ae574b3c2"} Jul 08 16:04:14.277466 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.277430 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" event={"ID":"c9fcff4a-b92c-4883-ae62-6a74b4f1e89d","Type":"ContainerStarted","Data":"4e19befae07e51d552d3a98ee35bd0e604eff1ed3ba9e54f6a1ee092a741fb2a"} Jul 08 16:04:14.277632 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.277610 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:14.279330 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.279312 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" Jul 08 16:04:14.291795 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.291754 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54bfc7ccf4-jrscw" podStartSLOduration=1.313680555 podStartE2EDuration="8.291741094s" totalImagesPullingTime="6.978060539s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:14.29099538 +0000 UTC m=+56.837840494" watchObservedRunningTime="2026-07-08 16:04:14.291741094 +0000 UTC m=+56.838586208" Jul 08 16:04:14.308946 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.308905 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-5cc85bb656-fldgz" podStartSLOduration=1.222274704 podStartE2EDuration="8.308893183s" totalImagesPullingTime="7.086618479s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:14.307997251 +0000 UTC m=+56.854842387" watchObservedRunningTime="2026-07-08 16:04:14.308893183 +0000 UTC m=+56.855738296" Jul 08 16:04:14.329540 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:14.329495 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" podStartSLOduration=1.303785951 podStartE2EDuration="8.329479252s" totalImagesPullingTime="7.025693301s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:04:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:14.328989524 +0000 UTC m=+56.875834638" watchObservedRunningTime="2026-07-08 16:04:14.329479252 +0000 UTC m=+56.876324366" Jul 08 16:04:16.555641 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:16.555599 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" podUID="609d11f4-ffaa-4b25-a75b-51ede4fd7f03" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 16:04:19.135060 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:19.135026 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g74wh" Jul 08 16:04:21.577009 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:21.576962 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:21.579296 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:21.579274 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/65d04972-2959-4b2b-b6b7-5f2ba538cc1d-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vmrh8\" (UID: \"65d04972-2959-4b2b-b6b7-5f2ba538cc1d\") " pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:21.677407 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:21.677375 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-2pd8l\"" Jul 08 16:04:21.684866 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:21.684847 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vmrh8" Jul 08 16:04:21.846236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:21.846085 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vmrh8"] Jul 08 16:04:21.848830 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:21.848802 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d04972_2959_4b2b_b6b7_5f2ba538cc1d.slice/crio-a36b079010a059a3812b9068770ab174182d90b6e4db5497b7433076e7699c53 WatchSource:0}: Error finding container a36b079010a059a3812b9068770ab174182d90b6e4db5497b7433076e7699c53: Status 404 returned error can't find the container with id a36b079010a059a3812b9068770ab174182d90b6e4db5497b7433076e7699c53 Jul 08 16:04:22.305318 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:22.305283 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmrh8" event={"ID":"65d04972-2959-4b2b-b6b7-5f2ba538cc1d","Type":"ContainerStarted","Data":"a36b079010a059a3812b9068770ab174182d90b6e4db5497b7433076e7699c53"} Jul 08 16:04:23.697174 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.697144 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:04:23.697542 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.697182 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:04:23.700508 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.700486 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 08 16:04:23.700508 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.700504 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 08 16:04:23.710216 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.710186 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d003ff65-18de-4f6a-91b8-5dd4c2db3e12-original-pull-secret\") pod \"global-pull-secret-syncer-wvsd9\" (UID: \"d003ff65-18de-4f6a-91b8-5dd4c2db3e12\") " pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:04:23.710308 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.710252 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8a6cf69c-8067-47bf-b481-a1691e96d9ad-metrics-certs\") pod \"network-metrics-daemon-8s9kq\" (UID: \"8a6cf69c-8067-47bf-b481-a1691e96d9ad\") " pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:04:23.797863 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.797808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:04:23.800760 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.800731 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbwtc\" (UniqueName: \"kubernetes.io/projected/d4b8a21c-b467-4c86-9973-0e1945fc5a20-kube-api-access-vbwtc\") pod \"network-check-target-gfhcb\" (UID: \"d4b8a21c-b467-4c86-9973-0e1945fc5a20\") " pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:04:23.925620 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.925586 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-pr5b4\"" Jul 08 16:04:23.933214 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.933192 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:04:23.937262 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.937234 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-wzt7p\"" Jul 08 16:04:23.944657 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.944632 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8s9kq" Jul 08 16:04:23.961627 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:23.961572 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-wvsd9" Jul 08 16:04:24.711034 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.710985 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:04:24.713583 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.713559 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"image-registry-d67db9954-tkv9c\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:04:24.812143 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.812106 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:04:24.814938 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.814894 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b6ea2261-c7e6-49b0-84fb-0a7a3c736f08-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-8x9d5\" (UID: \"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:04:24.883458 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.883423 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hjrqc\"" Jul 08 16:04:24.890608 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.890585 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:04:24.912770 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.912743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:04:24.912878 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.912781 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:24.912878 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.912802 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:04:24.912878 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.912826 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:24.913494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.913469 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e1292322-1790-4f8c-83ed-a18cae7ef53c-service-ca-bundle\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:24.915176 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.915151 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1292322-1790-4f8c-83ed-a18cae7ef53c-metrics-certs\") pod \"router-default-75f96688f5-4vp7w\" (UID: \"e1292322-1790-4f8c-83ed-a18cae7ef53c\") " pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:24.915290 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.915219 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/bfdb2476-acb0-483c-8612-85e59e2d5303-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-jhwzx\" (UID: \"bfdb2476-acb0-483c-8612-85e59e2d5303\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:04:24.915290 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.915251 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/93739528-2d92-4533-9351-eb673aad2c08-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-xpjfv\" (UID: \"93739528-2d92-4533-9351-eb673aad2c08\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:04:24.928339 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.928322 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-lwjvj\"" Jul 08 16:04:24.935900 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:24.935884 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" Jul 08 16:04:25.001937 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.001865 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-gp2gw\"" Jul 08 16:04:25.009538 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.009519 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:25.013978 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.013957 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:04:25.016204 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.016174 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54a18c97-d257-42c6-953a-115296d85fc5-metrics-tls\") pod \"dns-default-zhgft\" (UID: \"54a18c97-d257-42c6-953a-115296d85fc5\") " pod="openshift-dns/dns-default-zhgft" Jul 08 16:04:25.025101 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.025077 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-h58rq\"" Jul 08 16:04:25.032551 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.032535 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" Jul 08 16:04:25.071676 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.071656 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-vz8vc\"" Jul 08 16:04:25.078999 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.078984 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" Jul 08 16:04:25.131491 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.131469 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xz56b\"" Jul 08 16:04:25.138826 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:25.138804 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zhgft" Jul 08 16:04:26.510575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.510539 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8s9kq"] Jul 08 16:04:26.531512 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.531429 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv"] Jul 08 16:04:26.543050 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.543024 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx"] Jul 08 16:04:26.554244 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.554212 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" podUID="609d11f4-ffaa-4b25-a75b-51ede4fd7f03" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 16:04:26.625971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.625946 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gfhcb"] Jul 08 16:04:26.642456 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:26.642420 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6cf69c_8067_47bf_b481_a1691e96d9ad.slice/crio-ec9b2d72c9bfcaa4cdd978f3cc64b82415ca76778be8fd985fbf790f2ebd338e WatchSource:0}: Error finding container ec9b2d72c9bfcaa4cdd978f3cc64b82415ca76778be8fd985fbf790f2ebd338e: Status 404 returned error can't find the container with id ec9b2d72c9bfcaa4cdd978f3cc64b82415ca76778be8fd985fbf790f2ebd338e Jul 08 16:04:26.644634 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:26.644474 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfdb2476_acb0_483c_8612_85e59e2d5303.slice/crio-325ef9eb55bb08449f919db1396dc6f7775c120ade061d924ce96106a22ff63f WatchSource:0}: Error finding container 325ef9eb55bb08449f919db1396dc6f7775c120ade061d924ce96106a22ff63f: Status 404 returned error can't find the container with id 325ef9eb55bb08449f919db1396dc6f7775c120ade061d924ce96106a22ff63f Jul 08 16:04:26.645207 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:26.645184 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b8a21c_b467_4c86_9973_0e1945fc5a20.slice/crio-656815fa3f90ae51d9e8047fc98f9ac1d7969448fba87702a9f3c115bcc21246 WatchSource:0}: Error finding container 656815fa3f90ae51d9e8047fc98f9ac1d7969448fba87702a9f3c115bcc21246: Status 404 returned error can't find the container with id 656815fa3f90ae51d9e8047fc98f9ac1d7969448fba87702a9f3c115bcc21246 Jul 08 16:04:26.773447 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.773421 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-wvsd9"] Jul 08 16:04:26.776954 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.776887 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5"] Jul 08 16:04:26.778091 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.778036 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zhgft"] Jul 08 16:04:26.779769 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.779747 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-75f96688f5-4vp7w"] Jul 08 16:04:26.781165 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:26.781139 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d67db9954-tkv9c"] Jul 08 16:04:26.784989 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:26.784960 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1292322_1790_4f8c_83ed_a18cae7ef53c.slice/crio-9827d8dedfae7fa43d7482bb5c1e227de68e98a1cb657aa22f80c42a7b4d5b41 WatchSource:0}: Error finding container 9827d8dedfae7fa43d7482bb5c1e227de68e98a1cb657aa22f80c42a7b4d5b41: Status 404 returned error can't find the container with id 9827d8dedfae7fa43d7482bb5c1e227de68e98a1cb657aa22f80c42a7b4d5b41 Jul 08 16:04:26.785673 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:26.785650 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6ea2261_c7e6_49b0_84fb_0a7a3c736f08.slice/crio-099ee31fd40e1c92ae6c0614d311bfb903d6f09441f18aefc47e3c6fa7dee378 WatchSource:0}: Error finding container 099ee31fd40e1c92ae6c0614d311bfb903d6f09441f18aefc47e3c6fa7dee378: Status 404 returned error can't find the container with id 099ee31fd40e1c92ae6c0614d311bfb903d6f09441f18aefc47e3c6fa7dee378 Jul 08 16:04:26.786244 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:26.786221 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a18c97_d257_42c6_953a_115296d85fc5.slice/crio-97ca8c785fac7fcea08b7de37426a26ede160c5520e7db16d6d3eaf7f5fa1d3c WatchSource:0}: Error finding container 97ca8c785fac7fcea08b7de37426a26ede160c5520e7db16d6d3eaf7f5fa1d3c: Status 404 returned error can't find the container with id 97ca8c785fac7fcea08b7de37426a26ede160c5520e7db16d6d3eaf7f5fa1d3c Jul 08 16:04:26.787500 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:26.787479 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c32588_d44b_4083_8fbd_85f47cf30ea5.slice/crio-5b08810e3fd913266f3cb8539decd274bad8c1f9da6b13ea7a6f916f07007afd WatchSource:0}: Error finding container 5b08810e3fd913266f3cb8539decd274bad8c1f9da6b13ea7a6f916f07007afd: Status 404 returned error can't find the container with id 5b08810e3fd913266f3cb8539decd274bad8c1f9da6b13ea7a6f916f07007afd Jul 08 16:04:27.333946 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.333905 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmrh8" event={"ID":"65d04972-2959-4b2b-b6b7-5f2ba538cc1d","Type":"ContainerStarted","Data":"7d4e906cb06d7893ea19277c91b2dfb0ead7a85a62357d75bbb707afabfa643b"} Jul 08 16:04:27.335120 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.335087 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" event={"ID":"93739528-2d92-4533-9351-eb673aad2c08","Type":"ContainerStarted","Data":"8d54cc2bd9ed66141077321fa98ae85fa237f3e52d4fc001415fe80bbdb31f88"} Jul 08 16:04:27.336357 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.336334 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8s9kq" event={"ID":"8a6cf69c-8067-47bf-b481-a1691e96d9ad","Type":"ContainerStarted","Data":"ec9b2d72c9bfcaa4cdd978f3cc64b82415ca76778be8fd985fbf790f2ebd338e"} Jul 08 16:04:27.337603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.337560 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wvsd9" event={"ID":"d003ff65-18de-4f6a-91b8-5dd4c2db3e12","Type":"ContainerStarted","Data":"4901cdd19ef3020d38975ee8a95b982d6544594fb41adc8940fdfbcd20b16816"} Jul 08 16:04:27.338919 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.338893 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" event={"ID":"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08","Type":"ContainerStarted","Data":"099ee31fd40e1c92ae6c0614d311bfb903d6f09441f18aefc47e3c6fa7dee378"} Jul 08 16:04:27.340227 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.340181 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhgft" event={"ID":"54a18c97-d257-42c6-953a-115296d85fc5","Type":"ContainerStarted","Data":"97ca8c785fac7fcea08b7de37426a26ede160c5520e7db16d6d3eaf7f5fa1d3c"} Jul 08 16:04:27.341551 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.341523 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" event={"ID":"bfdb2476-acb0-483c-8612-85e59e2d5303","Type":"ContainerStarted","Data":"325ef9eb55bb08449f919db1396dc6f7775c120ade061d924ce96106a22ff63f"} Jul 08 16:04:27.343445 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.343419 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" event={"ID":"32c32588-d44b-4083-8fbd-85f47cf30ea5","Type":"ContainerStarted","Data":"b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691"} Jul 08 16:04:27.343879 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.343860 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" event={"ID":"32c32588-d44b-4083-8fbd-85f47cf30ea5","Type":"ContainerStarted","Data":"5b08810e3fd913266f3cb8539decd274bad8c1f9da6b13ea7a6f916f07007afd"} Jul 08 16:04:27.343963 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.343900 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:04:27.345486 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.345457 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75f96688f5-4vp7w" event={"ID":"e1292322-1790-4f8c-83ed-a18cae7ef53c","Type":"ContainerStarted","Data":"52b7bdceb841f0de1d49affab56bb799bc1014543d5374de99208e8b5cef2cba"} Jul 08 16:04:27.345589 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.345504 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-75f96688f5-4vp7w" event={"ID":"e1292322-1790-4f8c-83ed-a18cae7ef53c","Type":"ContainerStarted","Data":"9827d8dedfae7fa43d7482bb5c1e227de68e98a1cb657aa22f80c42a7b4d5b41"} Jul 08 16:04:27.347174 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.347151 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gfhcb" event={"ID":"d4b8a21c-b467-4c86-9973-0e1945fc5a20","Type":"ContainerStarted","Data":"b104afbb185d9d5f104b4b7e64abe3718ef6e78879e03971c425684d155ddb80"} Jul 08 16:04:27.347274 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.347192 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gfhcb" event={"ID":"d4b8a21c-b467-4c86-9973-0e1945fc5a20","Type":"ContainerStarted","Data":"656815fa3f90ae51d9e8047fc98f9ac1d7969448fba87702a9f3c115bcc21246"} Jul 08 16:04:27.347335 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.347319 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:04:27.373811 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.373757 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" podStartSLOduration=69.373740809 podStartE2EDuration="1m9.373740809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:27.373610074 +0000 UTC m=+69.920455189" watchObservedRunningTime="2026-07-08 16:04:27.373740809 +0000 UTC m=+69.920585900" Jul 08 16:04:27.416238 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.415963 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-75f96688f5-4vp7w" podStartSLOduration=60.4159292 podStartE2EDuration="1m0.4159292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:27.415745043 +0000 UTC m=+69.962590161" watchObservedRunningTime="2026-07-08 16:04:27.4159292 +0000 UTC m=+69.962774315" Jul 08 16:04:27.441097 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.441022 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gfhcb" podStartSLOduration=69.440993892 podStartE2EDuration="1m9.440993892s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:27.439864513 +0000 UTC m=+69.986709629" watchObservedRunningTime="2026-07-08 16:04:27.440993892 +0000 UTC m=+69.987839200" Jul 08 16:04:27.999237 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:27.999214 2563 scope.go:117] "RemoveContainer" containerID="03ac5e2497f793aad76d4f5b304f5575b57ae802e25c78e450f666b2828e46e4" Jul 08 16:04:28.009941 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:28.009913 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:28.035311 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:28.035056 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:28.353146 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:28.352896 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:28.355034 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:28.354822 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-75f96688f5-4vp7w" Jul 08 16:04:29.358927 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:29.358900 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:04:29.359389 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:29.359002 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" event={"ID":"1b73f265-f5e7-4aaa-a960-ea10101d0d55","Type":"ContainerStarted","Data":"17a4dbd00027e097c6d54edd148d5427c52a51c76de17bf5a95bc69e335cada7"} Jul 08 16:04:29.359570 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:29.359553 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:04:29.362180 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:29.362157 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmrh8" event={"ID":"65d04972-2959-4b2b-b6b7-5f2ba538cc1d","Type":"ContainerStarted","Data":"6f31b0f3b599a63c6ceafedfeb2c73f4ab1a344aa86f8b66ec94295f851e13a5"} Jul 08 16:04:29.381891 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:29.381833 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" podStartSLOduration=52.356100447 podStartE2EDuration="1m2.381811513s" totalImagesPullingTime="10.025711066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:29.379607622 +0000 UTC m=+71.926452740" watchObservedRunningTime="2026-07-08 16:04:29.381811513 +0000 UTC m=+71.928656629" Jul 08 16:04:30.183330 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.183300 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-wp8nf" Jul 08 16:04:30.367158 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.365132 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-685bfb5f64-gxwws"] Jul 08 16:04:30.381032 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.380997 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-gxwws" Jul 08 16:04:30.381521 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.381468 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-gxwws"] Jul 08 16:04:30.384270 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.383816 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-mvv57\"" Jul 08 16:04:30.384760 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.384538 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 08 16:04:30.384863 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.384790 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 08 16:04:30.463446 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.462980 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krr6z\" (UniqueName: \"kubernetes.io/projected/28f5cc45-4fcd-4a06-b6dd-00a1165763a3-kube-api-access-krr6z\") pod \"downloads-685bfb5f64-gxwws\" (UID: \"28f5cc45-4fcd-4a06-b6dd-00a1165763a3\") " pod="openshift-console/downloads-685bfb5f64-gxwws" Jul 08 16:04:30.563943 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.563900 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krr6z\" (UniqueName: \"kubernetes.io/projected/28f5cc45-4fcd-4a06-b6dd-00a1165763a3-kube-api-access-krr6z\") pod \"downloads-685bfb5f64-gxwws\" (UID: \"28f5cc45-4fcd-4a06-b6dd-00a1165763a3\") " pod="openshift-console/downloads-685bfb5f64-gxwws" Jul 08 16:04:30.575315 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.575281 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krr6z\" (UniqueName: \"kubernetes.io/projected/28f5cc45-4fcd-4a06-b6dd-00a1165763a3-kube-api-access-krr6z\") pod \"downloads-685bfb5f64-gxwws\" (UID: \"28f5cc45-4fcd-4a06-b6dd-00a1165763a3\") " pod="openshift-console/downloads-685bfb5f64-gxwws" Jul 08 16:04:30.693658 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:30.693626 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-685bfb5f64-gxwws" Jul 08 16:04:33.649368 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:33.649340 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-685bfb5f64-gxwws"] Jul 08 16:04:33.652136 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:33.651859 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f5cc45_4fcd_4a06_b6dd_00a1165763a3.slice/crio-92531bc2104e08a433f4d2b928144e56f5348f47cb431086239e564312a72db9 WatchSource:0}: Error finding container 92531bc2104e08a433f4d2b928144e56f5348f47cb431086239e564312a72db9: Status 404 returned error can't find the container with id 92531bc2104e08a433f4d2b928144e56f5348f47cb431086239e564312a72db9 Jul 08 16:04:34.393134 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:34.393095 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" event={"ID":"93739528-2d92-4533-9351-eb673aad2c08","Type":"ContainerStarted","Data":"f33484aa19bad9104c6bdbfbb460cc5f30b1cf8defe8536733cda1d2c6d37be9"} Jul 08 16:04:34.394856 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:34.394829 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8s9kq" event={"ID":"8a6cf69c-8067-47bf-b481-a1691e96d9ad","Type":"ContainerStarted","Data":"b36aac9020a25744cfac0316d8af3e1d522495c9a0d13bc2d637321cbc467ef0"} Jul 08 16:04:34.396633 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:34.396610 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" event={"ID":"b6ea2261-c7e6-49b0-84fb-0a7a3c736f08","Type":"ContainerStarted","Data":"7482a2c09c1501c27037d8b9fb0420e875dd86d5d7c470aa58a8f64000a9d461"} Jul 08 16:04:34.397996 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:34.397972 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-gxwws" event={"ID":"28f5cc45-4fcd-4a06-b6dd-00a1165763a3","Type":"ContainerStarted","Data":"92531bc2104e08a433f4d2b928144e56f5348f47cb431086239e564312a72db9"} Jul 08 16:04:34.399500 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:34.399471 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhgft" event={"ID":"54a18c97-d257-42c6-953a-115296d85fc5","Type":"ContainerStarted","Data":"daf6e88abb1d162e52cb5b9ca6b3f3fa0d3f336d4b73b61dd1510752756c898b"} Jul 08 16:04:34.401485 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:34.401465 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" event={"ID":"bfdb2476-acb0-483c-8612-85e59e2d5303","Type":"ContainerStarted","Data":"224218c508b13e42a74c6acb77a05de32360500352caa7adae22797e34fd4000"} Jul 08 16:04:34.416893 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:34.416835 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-8x9d5" podStartSLOduration=60.693752185 podStartE2EDuration="1m7.416822178s" totalImagesPullingTime="6.723069993s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:34.41594197 +0000 UTC m=+76.962787087" watchObservedRunningTime="2026-07-08 16:04:34.416822178 +0000 UTC m=+76.963667291" Jul 08 16:04:34.435284 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:34.435230 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-jhwzx" podStartSLOduration=60.636061621 podStartE2EDuration="1m7.435212836s" totalImagesPullingTime="6.799151215s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:34.43402622 +0000 UTC m=+76.980871325" watchObservedRunningTime="2026-07-08 16:04:34.435212836 +0000 UTC m=+76.982057954" Jul 08 16:04:35.408494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.408456 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" event={"ID":"93739528-2d92-4533-9351-eb673aad2c08","Type":"ContainerStarted","Data":"559f363f90e599113a717a8f8f1916ea79807b8efce03ef4d59b93b25ec2aeb8"} Jul 08 16:04:35.410406 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.410370 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8s9kq" event={"ID":"8a6cf69c-8067-47bf-b481-a1691e96d9ad","Type":"ContainerStarted","Data":"b57a071cbfe9c2449a34f8920e7bb88304e9a5388f633cb3ab1f9777e446a16e"} Jul 08 16:04:35.412257 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.412210 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-wvsd9" event={"ID":"d003ff65-18de-4f6a-91b8-5dd4c2db3e12","Type":"ContainerStarted","Data":"419a09ed07ae46675d92509812f24445e9b468c69151c96bbf5c8160d7841a3a"} Jul 08 16:04:35.414571 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.414545 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zhgft" event={"ID":"54a18c97-d257-42c6-953a-115296d85fc5","Type":"ContainerStarted","Data":"4e0a44a2319e0bbb052ca0b280368c875a8d2bf35ce937fadee2acb00307c3d1"} Jul 08 16:04:35.414868 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.414851 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zhgft" Jul 08 16:04:35.428253 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.428208 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-xpjfv" podStartSLOduration=61.641849676 podStartE2EDuration="1m8.428190704s" totalImagesPullingTime="6.786341028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:35.427858906 +0000 UTC m=+77.974704021" watchObservedRunningTime="2026-07-08 16:04:35.428190704 +0000 UTC m=+77.975035820" Jul 08 16:04:35.445654 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.445606 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8s9kq" podStartSLOduration=70.600213029 podStartE2EDuration="1m17.44559109s" totalImagesPullingTime="6.845378061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:35.444674162 +0000 UTC m=+77.991519278" watchObservedRunningTime="2026-07-08 16:04:35.44559109 +0000 UTC m=+77.992436203" Jul 08 16:04:35.461968 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.461913 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-wvsd9" podStartSLOduration=69.088811739 podStartE2EDuration="1m17.461894978s" totalImagesPullingTime="8.373083239s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:03:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:35.460263124 +0000 UTC m=+78.007108261" watchObservedRunningTime="2026-07-08 16:04:35.461894978 +0000 UTC m=+78.008740093" Jul 08 16:04:35.479055 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:35.478999 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zhgft" podStartSLOduration=36.753803366 podStartE2EDuration="43.478979761s" totalImagesPullingTime="6.725176395s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:03:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:35.478635253 +0000 UTC m=+78.025480367" watchObservedRunningTime="2026-07-08 16:04:35.478979761 +0000 UTC m=+78.025824875" Jul 08 16:04:36.422232 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:36.422185 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vmrh8" event={"ID":"65d04972-2959-4b2b-b6b7-5f2ba538cc1d","Type":"ContainerStarted","Data":"7bc0120ecd6a56cbb797c61f4d7c650b992a77475f9f556a71483f70b54f418e"} Jul 08 16:04:36.443384 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:36.443332 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vmrh8" podStartSLOduration=18.069858859 podStartE2EDuration="31.443314383s" totalImagesPullingTime="13.373455524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:04:05 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:36.442507994 +0000 UTC m=+78.989353112" watchObservedRunningTime="2026-07-08 16:04:36.443314383 +0000 UTC m=+78.990159539" Jul 08 16:04:36.553957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:36.553919 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" podUID="609d11f4-ffaa-4b25-a75b-51ede4fd7f03" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500" Jul 08 16:04:36.554153 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:36.553983 2563 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" Jul 08 16:04:36.554585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:36.554549 2563 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b0e68eb6ddd6c83731b71bd4a418096ec0dcc984870a4e250327b1e6722434eb"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" containerMessage="Container service-proxy failed liveness probe, will be restarted" Jul 08 16:04:36.554645 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:36.554615 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" podUID="609d11f4-ffaa-4b25-a75b-51ede4fd7f03" containerName="service-proxy" containerID="cri-o://b0e68eb6ddd6c83731b71bd4a418096ec0dcc984870a4e250327b1e6722434eb" gracePeriod=30 Jul 08 16:04:37.429488 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:37.429454 2563 generic.go:356] "Generic (PLEG): container finished" podID="609d11f4-ffaa-4b25-a75b-51ede4fd7f03" containerID="b0e68eb6ddd6c83731b71bd4a418096ec0dcc984870a4e250327b1e6722434eb" exitCode=2 Jul 08 16:04:37.429891 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:37.429518 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" event={"ID":"609d11f4-ffaa-4b25-a75b-51ede4fd7f03","Type":"ContainerDied","Data":"b0e68eb6ddd6c83731b71bd4a418096ec0dcc984870a4e250327b1e6722434eb"} Jul 08 16:04:37.429891 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:37.429572 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-88968469f-sfck6" event={"ID":"609d11f4-ffaa-4b25-a75b-51ede4fd7f03","Type":"ContainerStarted","Data":"f8c5d79ab773e3d83589ebfbae6702d013c2bdb98cbe5acb6e1e073a091c95e8"} Jul 08 16:04:44.897195 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:44.897036 2563 patch_prober.go:28] interesting pod/image-registry-d67db9954-tkv9c container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]} Jul 08 16:04:44.897195 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:44.897126 2563 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" podUID="32c32588-d44b-4083-8fbd-85f47cf30ea5" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 16:04:45.424734 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:45.424702 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zhgft" Jul 08 16:04:48.356534 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:48.356506 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:04:49.663545 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.663509 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-stpsb"] Jul 08 16:04:49.668095 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.667555 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.670766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.670516 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 08 16:04:49.670766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.670532 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 08 16:04:49.670766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.670519 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 08 16:04:49.670766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.670699 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-d2kxh\"" Jul 08 16:04:49.671264 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.671245 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 08 16:04:49.726782 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.726740 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-tls\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.726957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.726798 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.726957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.726826 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.726957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.726885 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-wtmp\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.726957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.726920 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f6570cf-85d1-47bc-a87e-d091b30452e0-metrics-client-ca\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.726957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.726945 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79nnp\" (UniqueName: \"kubernetes.io/projected/0f6570cf-85d1-47bc-a87e-d091b30452e0-kube-api-access-79nnp\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.727250 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.726991 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-textfile\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.727250 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.727040 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-root\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.727250 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.727084 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-sys\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827563 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-wtmp\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827612 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f6570cf-85d1-47bc-a87e-d091b30452e0-metrics-client-ca\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827730 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79nnp\" (UniqueName: \"kubernetes.io/projected/0f6570cf-85d1-47bc-a87e-d091b30452e0-kube-api-access-79nnp\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827764 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-wtmp\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827822 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-textfile\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827890 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-root\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827925 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-sys\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827956 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-tls\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.827988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.828029 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.828137 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-sys\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.828273 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0f6570cf-85d1-47bc-a87e-d091b30452e0-metrics-client-ca\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.828335 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0f6570cf-85d1-47bc-a87e-d091b30452e0-root\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:49.828418 2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.828421 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-textfile\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.828637 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:04:49.828481 2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-tls podName:0f6570cf-85d1-47bc-a87e-d091b30452e0 nodeName:}" failed. No retries permitted until 2026-07-08 16:04:50.328461993 +0000 UTC m=+92.875307088 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-tls") pod "node-exporter-stpsb" (UID: "0f6570cf-85d1-47bc-a87e-d091b30452e0") : secret "node-exporter-tls" not found Jul 08 16:04:49.829529 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.828940 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-accelerators-collector-config\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.831792 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.831740 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:49.839624 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:49.839577 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79nnp\" (UniqueName: \"kubernetes.io/projected/0f6570cf-85d1-47bc-a87e-d091b30452e0-kube-api-access-79nnp\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:50.333705 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:50.333663 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-tls\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:50.336592 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:50.336555 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0f6570cf-85d1-47bc-a87e-d091b30452e0-node-exporter-tls\") pod \"node-exporter-stpsb\" (UID: \"0f6570cf-85d1-47bc-a87e-d091b30452e0\") " pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:50.579505 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:50.579472 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-stpsb" Jul 08 16:04:54.882264 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:54.882214 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d67db9954-tkv9c"] Jul 08 16:04:56.841864 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:04:56.841838 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f6570cf_85d1_47bc_a87e_d091b30452e0.slice/crio-8dba1a923bb647bc2683fe728818d9bc04ed258aae3519c7031fce5be0cac044 WatchSource:0}: Error finding container 8dba1a923bb647bc2683fe728818d9bc04ed258aae3519c7031fce5be0cac044: Status 404 returned error can't find the container with id 8dba1a923bb647bc2683fe728818d9bc04ed258aae3519c7031fce5be0cac044 Jul 08 16:04:57.523618 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:57.523519 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-stpsb" event={"ID":"0f6570cf-85d1-47bc-a87e-d091b30452e0","Type":"ContainerStarted","Data":"8dba1a923bb647bc2683fe728818d9bc04ed258aae3519c7031fce5be0cac044"} Jul 08 16:04:57.525296 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:57.525262 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-685bfb5f64-gxwws" event={"ID":"28f5cc45-4fcd-4a06-b6dd-00a1165763a3","Type":"ContainerStarted","Data":"7988e23e116c64094c9a129b598751f54a8bcbc8bdcd121647c67405ee444953"} Jul 08 16:04:57.525665 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:57.525641 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-685bfb5f64-gxwws" Jul 08 16:04:57.527247 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:57.527205 2563 patch_prober.go:28] interesting pod/downloads-685bfb5f64-gxwws container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" start-of-body= Jul 08 16:04:57.527373 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:57.527263 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-685bfb5f64-gxwws" podUID="28f5cc45-4fcd-4a06-b6dd-00a1165763a3" containerName="download-server" probeResult="failure" output="Get \"http://10.132.0.25:8080/\": dial tcp 10.132.0.25:8080: connect: connection refused" Jul 08 16:04:57.547834 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:57.547784 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-685bfb5f64-gxwws" podStartSLOduration=3.988193641 podStartE2EDuration="27.547769431s" totalImagesPullingTime="23.55957579s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:04:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:57.547507034 +0000 UTC m=+100.094352149" watchObservedRunningTime="2026-07-08 16:04:57.547769431 +0000 UTC m=+100.094614566" Jul 08 16:04:58.355168 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:58.355084 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gfhcb" Jul 08 16:04:58.532376 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:58.532335 2563 generic.go:356] "Generic (PLEG): container finished" podID="0f6570cf-85d1-47bc-a87e-d091b30452e0" containerID="d07fe77ef7ab98c48adcd090efa7e5b96f3913578c8937a0ae4cc3fb0252837a" exitCode=0 Jul 08 16:04:58.532559 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:58.532424 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-stpsb" event={"ID":"0f6570cf-85d1-47bc-a87e-d091b30452e0","Type":"ContainerDied","Data":"d07fe77ef7ab98c48adcd090efa7e5b96f3913578c8937a0ae4cc3fb0252837a"} Jul 08 16:04:58.551348 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:58.551318 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-685bfb5f64-gxwws" Jul 08 16:04:59.540279 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:59.540239 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-stpsb" event={"ID":"0f6570cf-85d1-47bc-a87e-d091b30452e0","Type":"ContainerStarted","Data":"c3f71254c5cc0d6a7e7ff7a126c843b964971e04f8d2d93f39730765ba722310"} Jul 08 16:04:59.540760 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:59.540308 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-stpsb" event={"ID":"0f6570cf-85d1-47bc-a87e-d091b30452e0","Type":"ContainerStarted","Data":"c0a79ec425bc3c79688561ea430d38a90d3ae3ed009e9e6a088d78ab17d03471"} Jul 08 16:04:59.568243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:04:59.568176 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-stpsb" podStartSLOduration=9.357605842 podStartE2EDuration="10.568154964s" totalImagesPullingTime="1.132499234s" totalInitContainerRuntime="78.049888ms" isStatefulPod=true podCreationTimestamp="2026-07-08 16:04:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:04:59.565777648 +0000 UTC m=+102.112622759" watchObservedRunningTime="2026-07-08 16:04:59.568154964 +0000 UTC m=+102.115000079" Jul 08 16:05:19.909103 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:19.909030 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" podUID="32c32588-d44b-4083-8fbd-85f47cf30ea5" containerName="registry" containerID="cri-o://b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691" gracePeriod=30 Jul 08 16:05:20.187220 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.187155 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:05:20.205572 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.205546 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-image-registry-private-configuration\") pod \"32c32588-d44b-4083-8fbd-85f47cf30ea5\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " Jul 08 16:05:20.205702 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.205599 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-bound-sa-token\") pod \"32c32588-d44b-4083-8fbd-85f47cf30ea5\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " Jul 08 16:05:20.205702 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.205629 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") pod \"32c32588-d44b-4083-8fbd-85f47cf30ea5\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " Jul 08 16:05:20.205702 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.205651 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-certificates\") pod \"32c32588-d44b-4083-8fbd-85f47cf30ea5\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " Jul 08 16:05:20.205702 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.205674 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-trusted-ca\") pod \"32c32588-d44b-4083-8fbd-85f47cf30ea5\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " Jul 08 16:05:20.205702 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.205694 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-installation-pull-secrets\") pod \"32c32588-d44b-4083-8fbd-85f47cf30ea5\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " Jul 08 16:05:20.205934 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.205731 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2z5b\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-kube-api-access-h2z5b\") pod \"32c32588-d44b-4083-8fbd-85f47cf30ea5\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " Jul 08 16:05:20.205934 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.205758 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32c32588-d44b-4083-8fbd-85f47cf30ea5-ca-trust-extracted\") pod \"32c32588-d44b-4083-8fbd-85f47cf30ea5\" (UID: \"32c32588-d44b-4083-8fbd-85f47cf30ea5\") " Jul 08 16:05:20.206286 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.206169 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "32c32588-d44b-4083-8fbd-85f47cf30ea5" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 16:05:20.206400 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.206325 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "32c32588-d44b-4083-8fbd-85f47cf30ea5" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 16:05:20.208345 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.208317 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "32c32588-d44b-4083-8fbd-85f47cf30ea5" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:05:20.208472 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.208364 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "32c32588-d44b-4083-8fbd-85f47cf30ea5" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 16:05:20.208472 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.208413 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "32c32588-d44b-4083-8fbd-85f47cf30ea5" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:05:20.208581 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.208501 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-kube-api-access-h2z5b" (OuterVolumeSpecName: "kube-api-access-h2z5b") pod "32c32588-d44b-4083-8fbd-85f47cf30ea5" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5"). InnerVolumeSpecName "kube-api-access-h2z5b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:05:20.208581 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.208518 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "32c32588-d44b-4083-8fbd-85f47cf30ea5" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 16:05:20.215295 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.215267 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32c32588-d44b-4083-8fbd-85f47cf30ea5-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "32c32588-d44b-4083-8fbd-85f47cf30ea5" (UID: "32c32588-d44b-4083-8fbd-85f47cf30ea5"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:05:20.307174 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.307134 2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-image-registry-private-configuration\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:05:20.307174 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.307166 2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-bound-sa-token\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:05:20.307174 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.307176 2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-tls\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:05:20.307398 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.307188 2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-registry-certificates\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:05:20.307398 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.307201 2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/32c32588-d44b-4083-8fbd-85f47cf30ea5-trusted-ca\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:05:20.307398 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.307210 2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/32c32588-d44b-4083-8fbd-85f47cf30ea5-installation-pull-secrets\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:05:20.307398 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.307217 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2z5b\" (UniqueName: \"kubernetes.io/projected/32c32588-d44b-4083-8fbd-85f47cf30ea5-kube-api-access-h2z5b\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:05:20.307398 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.307225 2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/32c32588-d44b-4083-8fbd-85f47cf30ea5-ca-trust-extracted\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:05:20.626694 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.626609 2563 generic.go:356] "Generic (PLEG): container finished" podID="32c32588-d44b-4083-8fbd-85f47cf30ea5" containerID="b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691" exitCode=0 Jul 08 16:05:20.626694 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.626657 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" Jul 08 16:05:20.626694 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.626674 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" event={"ID":"32c32588-d44b-4083-8fbd-85f47cf30ea5","Type":"ContainerDied","Data":"b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691"} Jul 08 16:05:20.626944 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.626724 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d67db9954-tkv9c" event={"ID":"32c32588-d44b-4083-8fbd-85f47cf30ea5","Type":"ContainerDied","Data":"5b08810e3fd913266f3cb8539decd274bad8c1f9da6b13ea7a6f916f07007afd"} Jul 08 16:05:20.626944 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.626750 2563 scope.go:117] "RemoveContainer" containerID="b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691" Jul 08 16:05:20.635619 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.635600 2563 scope.go:117] "RemoveContainer" containerID="b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691" Jul 08 16:05:20.635859 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:05:20.635838 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691\": container with ID starting with b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691 not found: ID does not exist" containerID="b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691" Jul 08 16:05:20.635925 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.635867 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691"} err="failed to get container status \"b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691\": rpc error: code = NotFound desc = could not find container \"b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691\": container with ID starting with b7caf23da11d3475c27b3205e3c55f5f1f0276cbcd240ecf8e9d6be62fa8f691 not found: ID does not exist" Jul 08 16:05:20.648886 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.648860 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-d67db9954-tkv9c"] Jul 08 16:05:20.653923 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:20.653903 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-d67db9954-tkv9c"] Jul 08 16:05:22.001658 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:22.001622 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c32588-d44b-4083-8fbd-85f47cf30ea5" path="/var/lib/kubelet/pods/32c32588-d44b-4083-8fbd-85f47cf30ea5/volumes" Jul 08 16:05:29.661656 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:29.661620 2563 generic.go:356] "Generic (PLEG): container finished" podID="d0721a8f-4745-41d6-944c-d4ab9a293612" containerID="4d253a5e88816b75b565e6e05e89ef66ceef1ca7f474c6fc7a4fa6368074309f" exitCode=0 Jul 08 16:05:29.662056 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:29.661665 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" event={"ID":"d0721a8f-4745-41d6-944c-d4ab9a293612","Type":"ContainerDied","Data":"4d253a5e88816b75b565e6e05e89ef66ceef1ca7f474c6fc7a4fa6368074309f"} Jul 08 16:05:29.662056 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:29.661968 2563 scope.go:117] "RemoveContainer" containerID="4d253a5e88816b75b565e6e05e89ef66ceef1ca7f474c6fc7a4fa6368074309f" Jul 08 16:05:30.666247 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:30.666215 2563 generic.go:356] "Generic (PLEG): container finished" podID="76bba082-a1e8-4954-9011-9c6209aa5ae6" containerID="50ce60d41038fb78e33d58c18b4658a26ee889f93b573efb228c4ef9ae12164c" exitCode=0 Jul 08 16:05:30.666664 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:30.666280 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" event={"ID":"76bba082-a1e8-4954-9011-9c6209aa5ae6","Type":"ContainerDied","Data":"50ce60d41038fb78e33d58c18b4658a26ee889f93b573efb228c4ef9ae12164c"} Jul 08 16:05:30.666664 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:30.666615 2563 scope.go:117] "RemoveContainer" containerID="50ce60d41038fb78e33d58c18b4658a26ee889f93b573efb228c4ef9ae12164c" Jul 08 16:05:30.668105 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:30.668084 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-mv777" event={"ID":"d0721a8f-4745-41d6-944c-d4ab9a293612","Type":"ContainerStarted","Data":"c650c3305951b8ade006e93fc07700b5ebb5c24b61d843c51964a0f62bfdd6d2"} Jul 08 16:05:31.411014 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:31.410984 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f96688f5-4vp7w_e1292322-1790-4f8c-83ed-a18cae7ef53c/router/0.log" Jul 08 16:05:31.419704 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:31.419678 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qthjk_49d96c6c-2207-4d00-ab5e-bf123f948a0f/serve-healthcheck-canary/0.log" Jul 08 16:05:31.673102 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:31.673017 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-vqmfn" event={"ID":"76bba082-a1e8-4954-9011-9c6209aa5ae6","Type":"ContainerStarted","Data":"bab92b1e2d78efa95687aa6b913ec966c4fb32c73566771da32bfde1b2440e93"} Jul 08 16:05:34.685770 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:34.685735 2563 generic.go:356] "Generic (PLEG): container finished" podID="7f660b8b-1a38-4e75-b353-11842c393317" containerID="d1e1a27b5d877ae39c9dab2926b85f79212b3be6d4dd513e4fb36aeb8cb2b26e" exitCode=0 Jul 08 16:05:34.686175 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:34.685812 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" event={"ID":"7f660b8b-1a38-4e75-b353-11842c393317","Type":"ContainerDied","Data":"d1e1a27b5d877ae39c9dab2926b85f79212b3be6d4dd513e4fb36aeb8cb2b26e"} Jul 08 16:05:34.686230 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:34.686216 2563 scope.go:117] "RemoveContainer" containerID="d1e1a27b5d877ae39c9dab2926b85f79212b3be6d4dd513e4fb36aeb8cb2b26e" Jul 08 16:05:35.691287 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:05:35.691253 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-tdhsf" event={"ID":"7f660b8b-1a38-4e75-b353-11842c393317","Type":"ContainerStarted","Data":"981c32229c7d8a071e53b57e5a653da01793873fd0df215c2642d59b4a6906a4"} Jul 08 16:08:17.896038 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:17.896005 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:08:17.901263 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:17.901246 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:08:17.904747 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:17.904731 2563 kubelet.go:1648] "Image garbage collection succeeded" Jul 08 16:08:34.220216 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.220182 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5"] Jul 08 16:08:34.221853 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.220515 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c32588-d44b-4083-8fbd-85f47cf30ea5" containerName="registry" Jul 08 16:08:34.221853 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.220526 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c32588-d44b-4083-8fbd-85f47cf30ea5" containerName="registry" Jul 08 16:08:34.221853 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.220586 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c32588-d44b-4083-8fbd-85f47cf30ea5" containerName="registry" Jul 08 16:08:34.222553 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.222538 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.225728 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.225710 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 08 16:08:34.225853 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.225737 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 08 16:08:34.227953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.227929 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-gr2cb\"" Jul 08 16:08:34.228151 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.228104 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 08 16:08:34.228319 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.228299 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 08 16:08:34.252014 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.251986 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5"] Jul 08 16:08:34.252445 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.252425 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-manifests\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.252517 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.252452 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.252517 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.252475 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fpbqd\" (UniqueName: \"kubernetes.io/projected/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-kube-api-access-fpbqd\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.252517 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.252498 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-tmp\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.252627 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.252550 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.353632 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.353598 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-manifests\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.353835 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.353636 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.353835 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.353672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fpbqd\" (UniqueName: \"kubernetes.io/projected/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-kube-api-access-fpbqd\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.353835 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.353706 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-tmp\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.353835 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.353734 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.354089 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.354042 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-manifests\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.356007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.355973 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-tmp\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.356157 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.356105 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-webhook-cert\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.356157 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.356134 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-apiservice-cert\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.363510 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.363486 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fpbqd\" (UniqueName: \"kubernetes.io/projected/8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4-kube-api-access-fpbqd\") pod \"opendatahub-operator-controller-manager-bf8d94745-bqdc5\" (UID: \"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4\") " pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.533234 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.533146 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:34.672399 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.672363 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5"] Jul 08 16:08:34.675817 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:08:34.675788 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b0fdb5b_7ab1_456e_bc26_2f9bc5a7b3a4.slice/crio-a0ec54d22e1eb61c0291c769a2ecd483933c75045348caa6ec36040338cd8cfe WatchSource:0}: Error finding container a0ec54d22e1eb61c0291c769a2ecd483933c75045348caa6ec36040338cd8cfe: Status 404 returned error can't find the container with id a0ec54d22e1eb61c0291c769a2ecd483933c75045348caa6ec36040338cd8cfe Jul 08 16:08:34.677527 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:34.677506 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 16:08:35.391766 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:35.391729 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" event={"ID":"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4","Type":"ContainerStarted","Data":"a0ec54d22e1eb61c0291c769a2ecd483933c75045348caa6ec36040338cd8cfe"} Jul 08 16:08:38.407679 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:38.407639 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" event={"ID":"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4","Type":"ContainerStarted","Data":"1c06dae295f7413233a37a95b26f1c14b7799029623b8593995733eedd1653fe"} Jul 08 16:08:39.413491 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:39.413456 2563 generic.go:356] "Generic (PLEG): container finished" podID="8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4" containerID="1c06dae295f7413233a37a95b26f1c14b7799029623b8593995733eedd1653fe" exitCode=0 Jul 08 16:08:39.413855 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:39.413548 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" event={"ID":"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4","Type":"ContainerDied","Data":"1c06dae295f7413233a37a95b26f1c14b7799029623b8593995733eedd1653fe"} Jul 08 16:08:40.419955 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:40.419912 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" event={"ID":"8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4","Type":"ContainerStarted","Data":"361a1532831dea7eab32be76a196e1016fd7b9e4efaecbf7ea717d1bdccd5d5c"} Jul 08 16:08:40.420367 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:40.420051 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:08:40.445586 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:40.445537 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" podStartSLOduration=1.846712239 podStartE2EDuration="6.445523343s" totalImagesPullingTime="3.570239108s" totalInitContainerRuntime="1.028571996s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:08:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:08:40.443117648 +0000 UTC m=+322.989962761" watchObservedRunningTime="2026-07-08 16:08:40.445523343 +0000 UTC m=+322.992368458" Jul 08 16:08:51.427425 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:08:51.427391 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-bf8d94745-bqdc5" Jul 08 16:09:02.953393 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.953343 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp"] Jul 08 16:09:02.964076 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.964026 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:02.969044 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.968515 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-d5gqg\"" Jul 08 16:09:02.969044 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.968778 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 08 16:09:02.969044 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.968925 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 08 16:09:02.969044 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.968946 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 08 16:09:02.969044 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.968980 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 08 16:09:02.969438 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.969204 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 08 16:09:02.977368 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:02.977343 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp"] Jul 08 16:09:03.090953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.090921 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-manager-config\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.091140 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.091027 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd756\" (UniqueName: \"kubernetes.io/projected/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-kube-api-access-gd756\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.091140 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.091101 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-cert\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.091140 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.091119 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-metrics-cert\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.192477 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.192436 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd756\" (UniqueName: \"kubernetes.io/projected/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-kube-api-access-gd756\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.192654 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.192490 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-cert\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.192654 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.192508 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-metrics-cert\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.192654 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.192530 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-manager-config\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.193179 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.193157 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-manager-config\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.194960 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.194940 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-cert\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.195083 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.194993 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-metrics-cert\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.203793 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.203723 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd756\" (UniqueName: \"kubernetes.io/projected/6c5a02b2-00f4-4437-a1d8-4519a49dca9f-kube-api-access-gd756\") pod \"lws-controller-manager-7bf555d8d4-45vzp\" (UID: \"6c5a02b2-00f4-4437-a1d8-4519a49dca9f\") " pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.278797 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.278762 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:03.409663 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.409632 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp"] Jul 08 16:09:03.412534 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:09:03.412508 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c5a02b2_00f4_4437_a1d8_4519a49dca9f.slice/crio-a2536f6d6f35099648bcd8a694ca09d5978c71990fc9f5ff9505e2fd74d2858f WatchSource:0}: Error finding container a2536f6d6f35099648bcd8a694ca09d5978c71990fc9f5ff9505e2fd74d2858f: Status 404 returned error can't find the container with id a2536f6d6f35099648bcd8a694ca09d5978c71990fc9f5ff9505e2fd74d2858f Jul 08 16:09:03.520777 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:03.520693 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" event={"ID":"6c5a02b2-00f4-4437-a1d8-4519a49dca9f","Type":"ContainerStarted","Data":"a2536f6d6f35099648bcd8a694ca09d5978c71990fc9f5ff9505e2fd74d2858f"} Jul 08 16:09:07.545907 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:07.545855 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" event={"ID":"6c5a02b2-00f4-4437-a1d8-4519a49dca9f","Type":"ContainerStarted","Data":"e91d725d63c17ada379d576f596460dc51d656b843fc2d9611d20363194380d5"} Jul 08 16:09:07.546346 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:07.545950 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:09:07.563643 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:07.563593 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" podStartSLOduration=1.883127027 podStartE2EDuration="5.563580282s" totalImagesPullingTime="3.680453255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:09:07.562357854 +0000 UTC m=+350.109202975" watchObservedRunningTime="2026-07-08 16:09:07.563580282 +0000 UTC m=+350.110425395" Jul 08 16:09:18.553401 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:09:18.553331 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-7bf555d8d4-45vzp" Jul 08 16:10:35.157025 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.156992 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w"] Jul 08 16:10:35.159321 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.159304 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" Jul 08 16:10:35.163921 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.163900 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\"" Jul 08 16:10:35.165046 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.165023 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kp75r\"" Jul 08 16:10:35.165046 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.165040 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 08 16:10:35.165220 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.165088 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 08 16:10:35.176818 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.176798 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w"] Jul 08 16:10:35.330995 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.330948 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4w4l\" (UniqueName: \"kubernetes.io/projected/d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5-kube-api-access-m4w4l\") pod \"dns-operator-controller-manager-864dd78c4c-bf74w\" (UID: \"d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" Jul 08 16:10:35.431478 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.431367 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4w4l\" (UniqueName: \"kubernetes.io/projected/d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5-kube-api-access-m4w4l\") pod \"dns-operator-controller-manager-864dd78c4c-bf74w\" (UID: \"d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" Jul 08 16:10:35.446468 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.446436 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4w4l\" (UniqueName: \"kubernetes.io/projected/d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5-kube-api-access-m4w4l\") pod \"dns-operator-controller-manager-864dd78c4c-bf74w\" (UID: \"d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" Jul 08 16:10:35.469610 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.469577 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" Jul 08 16:10:35.630454 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.630408 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w"] Jul 08 16:10:35.632824 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:10:35.632796 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bbaa9a_aa6b_4a9b_82e3_68feff48f2b5.slice/crio-6e671bf5639b2cc65be1ccde6c5b907f61a58d4d1fba2ae906dc927fc016c0b7 WatchSource:0}: Error finding container 6e671bf5639b2cc65be1ccde6c5b907f61a58d4d1fba2ae906dc927fc016c0b7: Status 404 returned error can't find the container with id 6e671bf5639b2cc65be1ccde6c5b907f61a58d4d1fba2ae906dc927fc016c0b7 Jul 08 16:10:35.920727 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:35.920693 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" event={"ID":"d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5","Type":"ContainerStarted","Data":"6e671bf5639b2cc65be1ccde6c5b907f61a58d4d1fba2ae906dc927fc016c0b7"} Jul 08 16:10:38.935964 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:38.935929 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" event={"ID":"d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5","Type":"ContainerStarted","Data":"c942c5f3d7698f4d8b4a0d6419d77b87b6142469649b06a5d0f0c61ca9babfaa"} Jul 08 16:10:38.936344 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:38.936043 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" Jul 08 16:10:38.958216 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:38.958175 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" podStartSLOduration=1.153625559 podStartE2EDuration="3.958161094s" totalImagesPullingTime="2.804535535s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:10:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:10:38.956008061 +0000 UTC m=+441.502853175" watchObservedRunningTime="2026-07-08 16:10:38.958161094 +0000 UTC m=+441.505006208" Jul 08 16:10:39.601729 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.601692 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj"] Jul 08 16:10:39.605394 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.605374 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:39.608295 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.608276 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hdf7b\"" Jul 08 16:10:39.628817 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.628793 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj"] Jul 08 16:10:39.767042 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.766983 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvx6\" (UniqueName: \"kubernetes.io/projected/d305d73a-09d7-49a4-b6a7-e24701b0a876-kube-api-access-jpvx6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\" (UID: \"d305d73a-09d7-49a4-b6a7-e24701b0a876\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:39.767229 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.767087 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d305d73a-09d7-49a4-b6a7-e24701b0a876-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\" (UID: \"d305d73a-09d7-49a4-b6a7-e24701b0a876\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:39.867771 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.867687 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d305d73a-09d7-49a4-b6a7-e24701b0a876-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\" (UID: \"d305d73a-09d7-49a4-b6a7-e24701b0a876\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:39.867771 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.867743 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvx6\" (UniqueName: \"kubernetes.io/projected/d305d73a-09d7-49a4-b6a7-e24701b0a876-kube-api-access-jpvx6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\" (UID: \"d305d73a-09d7-49a4-b6a7-e24701b0a876\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:39.868099 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.868047 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d305d73a-09d7-49a4-b6a7-e24701b0a876-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\" (UID: \"d305d73a-09d7-49a4-b6a7-e24701b0a876\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:39.876738 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.876709 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvx6\" (UniqueName: \"kubernetes.io/projected/d305d73a-09d7-49a4-b6a7-e24701b0a876-kube-api-access-jpvx6\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\" (UID: \"d305d73a-09d7-49a4-b6a7-e24701b0a876\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:39.917653 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:39.917608 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:40.066310 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:40.059780 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj"] Jul 08 16:10:40.947760 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:40.947687 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" event={"ID":"d305d73a-09d7-49a4-b6a7-e24701b0a876","Type":"ContainerStarted","Data":"aa1a8ca9f91670f47a5ea4168e598412a2f612f1168af5cbb2027634857c79f8"} Jul 08 16:10:48.984581 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:48.984540 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" event={"ID":"d305d73a-09d7-49a4-b6a7-e24701b0a876","Type":"ContainerStarted","Data":"439fe7cc0dccf8f4a568bbb2233dfbc7b3416804430409642144b9e7b0069dce"} Jul 08 16:10:48.984974 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:48.984683 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:10:49.011957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:49.011906 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" podStartSLOduration=1.655735505 podStartE2EDuration="10.011890226s" totalImagesPullingTime="8.356154721s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:10:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:10:49.010916966 +0000 UTC m=+451.557762082" watchObservedRunningTime="2026-07-08 16:10:49.011890226 +0000 UTC m=+451.558735340" Jul 08 16:10:49.944132 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:49.944099 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-bf74w" Jul 08 16:10:59.992627 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:10:59.992595 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:11:01.852650 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.852619 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj"] Jul 08 16:11:01.853152 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.852790 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" podUID="d305d73a-09d7-49a4-b6a7-e24701b0a876" containerName="manager" containerID="cri-o://439fe7cc0dccf8f4a568bbb2233dfbc7b3416804430409642144b9e7b0069dce" gracePeriod=2 Jul 08 16:11:01.868175 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.868143 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj"] Jul 08 16:11:01.881308 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:11:01.881271 2563 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"d305d73a-09d7-49a4-b6a7-e24701b0a876\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\": pods \"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:11:01.882896 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.882871 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv"] Jul 08 16:11:01.883303 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.883280 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d305d73a-09d7-49a4-b6a7-e24701b0a876" containerName="manager" Jul 08 16:11:01.883303 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.883303 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d305d73a-09d7-49a4-b6a7-e24701b0a876" containerName="manager" Jul 08 16:11:01.883458 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.883389 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d305d73a-09d7-49a4-b6a7-e24701b0a876" containerName="manager" Jul 08 16:11:01.886687 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.886668 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:01.908333 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.908307 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv"] Jul 08 16:11:01.927511 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:11:01.927476 2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6zsgj\" is forbidden: User \"system:node:ip-10-0-141-121.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-121.ec2.internal' and this object" podUID="d305d73a-09d7-49a4-b6a7-e24701b0a876" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:11:01.951061 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.951026 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c90e4-15c2-4086-bc40-a99a94fe862c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rl6pv\" (UID: \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:01.951257 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:01.951096 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn5kp\" (UniqueName: \"kubernetes.io/projected/0c9c90e4-15c2-4086-bc40-a99a94fe862c-kube-api-access-bn5kp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rl6pv\" (UID: \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:02.045969 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.045939 2563 generic.go:356] "Generic (PLEG): container finished" podID="d305d73a-09d7-49a4-b6a7-e24701b0a876" containerID="439fe7cc0dccf8f4a568bbb2233dfbc7b3416804430409642144b9e7b0069dce" exitCode=0 Jul 08 16:11:02.052258 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.052229 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c90e4-15c2-4086-bc40-a99a94fe862c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rl6pv\" (UID: \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:02.052428 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.052262 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn5kp\" (UniqueName: \"kubernetes.io/projected/0c9c90e4-15c2-4086-bc40-a99a94fe862c-kube-api-access-bn5kp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rl6pv\" (UID: \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:02.052678 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.052657 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c90e4-15c2-4086-bc40-a99a94fe862c-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rl6pv\" (UID: \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:02.077725 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.077695 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn5kp\" (UniqueName: \"kubernetes.io/projected/0c9c90e4-15c2-4086-bc40-a99a94fe862c-kube-api-access-bn5kp\") pod \"kuadrant-operator-controller-manager-8684c9ff49-rl6pv\" (UID: \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:02.099191 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.099164 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:11:02.153371 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.153344 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpvx6\" (UniqueName: \"kubernetes.io/projected/d305d73a-09d7-49a4-b6a7-e24701b0a876-kube-api-access-jpvx6\") pod \"d305d73a-09d7-49a4-b6a7-e24701b0a876\" (UID: \"d305d73a-09d7-49a4-b6a7-e24701b0a876\") " Jul 08 16:11:02.153508 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.153385 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d305d73a-09d7-49a4-b6a7-e24701b0a876-extensions-socket-volume\") pod \"d305d73a-09d7-49a4-b6a7-e24701b0a876\" (UID: \"d305d73a-09d7-49a4-b6a7-e24701b0a876\") " Jul 08 16:11:02.153900 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.153877 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d305d73a-09d7-49a4-b6a7-e24701b0a876-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d305d73a-09d7-49a4-b6a7-e24701b0a876" (UID: "d305d73a-09d7-49a4-b6a7-e24701b0a876"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:11:02.155547 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.155527 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d305d73a-09d7-49a4-b6a7-e24701b0a876-kube-api-access-jpvx6" (OuterVolumeSpecName: "kube-api-access-jpvx6") pod "d305d73a-09d7-49a4-b6a7-e24701b0a876" (UID: "d305d73a-09d7-49a4-b6a7-e24701b0a876"). InnerVolumeSpecName "kube-api-access-jpvx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:11:02.228636 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.228600 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:02.254668 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.254636 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvx6\" (UniqueName: \"kubernetes.io/projected/d305d73a-09d7-49a4-b6a7-e24701b0a876-kube-api-access-jpvx6\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:11:02.254668 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.254669 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d305d73a-09d7-49a4-b6a7-e24701b0a876-extensions-socket-volume\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:11:02.371710 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:02.371673 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv"] Jul 08 16:11:02.374367 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:11:02.374334 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9c90e4_15c2_4086_bc40_a99a94fe862c.slice/crio-6cfc6c049aedc29ffee5e37a746f8d558e78a85a2e09471c8999b5fb9724753c WatchSource:0}: Error finding container 6cfc6c049aedc29ffee5e37a746f8d558e78a85a2e09471c8999b5fb9724753c: Status 404 returned error can't find the container with id 6cfc6c049aedc29ffee5e37a746f8d558e78a85a2e09471c8999b5fb9724753c Jul 08 16:11:03.051759 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:03.051725 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" event={"ID":"0c9c90e4-15c2-4086-bc40-a99a94fe862c","Type":"ContainerStarted","Data":"093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b"} Jul 08 16:11:03.052259 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:03.051784 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" event={"ID":"0c9c90e4-15c2-4086-bc40-a99a94fe862c","Type":"ContainerStarted","Data":"6cfc6c049aedc29ffee5e37a746f8d558e78a85a2e09471c8999b5fb9724753c"} Jul 08 16:11:03.052259 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:03.051842 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:03.053113 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:03.053090 2563 scope.go:117] "RemoveContainer" containerID="439fe7cc0dccf8f4a568bbb2233dfbc7b3416804430409642144b9e7b0069dce" Jul 08 16:11:03.053113 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:03.053106 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6zsgj" Jul 08 16:11:03.075925 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:03.075877 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" podStartSLOduration=2.075861922 podStartE2EDuration="2.075861922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:11:03.073946096 +0000 UTC m=+465.620791209" watchObservedRunningTime="2026-07-08 16:11:03.075861922 +0000 UTC m=+465.622707035" Jul 08 16:11:04.002522 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:04.002477 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d305d73a-09d7-49a4-b6a7-e24701b0a876" path="/var/lib/kubelet/pods/d305d73a-09d7-49a4-b6a7-e24701b0a876/volumes" Jul 08 16:11:14.060677 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:14.060645 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:30.817042 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:30.817008 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv"] Jul 08 16:11:30.817459 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:30.817247 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" podUID="0c9c90e4-15c2-4086-bc40-a99a94fe862c" containerName="manager" containerID="cri-o://093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b" gracePeriod=10 Jul 08 16:11:31.042250 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.042223 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk"] Jul 08 16:11:31.044357 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.044341 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:31.058601 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.058580 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:31.065788 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.065759 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk"] Jul 08 16:11:31.176137 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.176109 2563 generic.go:356] "Generic (PLEG): container finished" podID="0c9c90e4-15c2-4086-bc40-a99a94fe862c" containerID="093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b" exitCode=0 Jul 08 16:11:31.176291 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.176154 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" Jul 08 16:11:31.176291 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.176192 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" event={"ID":"0c9c90e4-15c2-4086-bc40-a99a94fe862c","Type":"ContainerDied","Data":"093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b"} Jul 08 16:11:31.176291 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.176242 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv" event={"ID":"0c9c90e4-15c2-4086-bc40-a99a94fe862c","Type":"ContainerDied","Data":"6cfc6c049aedc29ffee5e37a746f8d558e78a85a2e09471c8999b5fb9724753c"} Jul 08 16:11:31.176291 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.176264 2563 scope.go:117] "RemoveContainer" containerID="093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b" Jul 08 16:11:31.187435 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.186121 2563 scope.go:117] "RemoveContainer" containerID="093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b" Jul 08 16:11:31.187435 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:11:31.186430 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b\": container with ID starting with 093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b not found: ID does not exist" containerID="093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b" Jul 08 16:11:31.187435 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.186476 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b"} err="failed to get container status \"093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b\": rpc error: code = NotFound desc = could not find container \"093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b\": container with ID starting with 093607e92c5bfd8a622c9a58ca433b8c10eccdf311d479b7ea382c0a9d242f4b not found: ID does not exist" Jul 08 16:11:31.202590 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.202571 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c90e4-15c2-4086-bc40-a99a94fe862c-extensions-socket-volume\") pod \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\" (UID: \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\") " Jul 08 16:11:31.202664 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.202639 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bn5kp\" (UniqueName: \"kubernetes.io/projected/0c9c90e4-15c2-4086-bc40-a99a94fe862c-kube-api-access-bn5kp\") pod \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\" (UID: \"0c9c90e4-15c2-4086-bc40-a99a94fe862c\") " Jul 08 16:11:31.202741 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.202727 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7k9jk\" (UID: \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:31.202793 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.202756 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfmls\" (UniqueName: \"kubernetes.io/projected/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-kube-api-access-cfmls\") pod \"kuadrant-operator-controller-manager-86878c769f-7k9jk\" (UID: \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:31.202919 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.202897 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c9c90e4-15c2-4086-bc40-a99a94fe862c-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0c9c90e4-15c2-4086-bc40-a99a94fe862c" (UID: "0c9c90e4-15c2-4086-bc40-a99a94fe862c"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:11:31.204642 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.204625 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c9c90e4-15c2-4086-bc40-a99a94fe862c-kube-api-access-bn5kp" (OuterVolumeSpecName: "kube-api-access-bn5kp") pod "0c9c90e4-15c2-4086-bc40-a99a94fe862c" (UID: "0c9c90e4-15c2-4086-bc40-a99a94fe862c"). InnerVolumeSpecName "kube-api-access-bn5kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:11:31.303787 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.303750 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7k9jk\" (UID: \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:31.303787 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.303796 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfmls\" (UniqueName: \"kubernetes.io/projected/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-kube-api-access-cfmls\") pod \"kuadrant-operator-controller-manager-86878c769f-7k9jk\" (UID: \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:31.303977 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.303847 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bn5kp\" (UniqueName: \"kubernetes.io/projected/0c9c90e4-15c2-4086-bc40-a99a94fe862c-kube-api-access-bn5kp\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:11:31.303977 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.303857 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0c9c90e4-15c2-4086-bc40-a99a94fe862c-extensions-socket-volume\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:11:31.304142 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.304121 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-7k9jk\" (UID: \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:31.314924 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.314892 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfmls\" (UniqueName: \"kubernetes.io/projected/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-kube-api-access-cfmls\") pod \"kuadrant-operator-controller-manager-86878c769f-7k9jk\" (UID: \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:31.366858 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.366828 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:31.498664 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.498638 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk"] Jul 08 16:11:31.500487 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:11:31.500458 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaed6c1ce_dcfb_46a9_968b_7009c4d3a4a7.slice/crio-668a27bfbcce0f6a5d7257c6d494075482e58e4077dd0337f26aa0c5f1e713b4 WatchSource:0}: Error finding container 668a27bfbcce0f6a5d7257c6d494075482e58e4077dd0337f26aa0c5f1e713b4: Status 404 returned error can't find the container with id 668a27bfbcce0f6a5d7257c6d494075482e58e4077dd0337f26aa0c5f1e713b4 Jul 08 16:11:31.503403 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.503381 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv"] Jul 08 16:11:31.508638 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:31.508619 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-rl6pv"] Jul 08 16:11:32.001662 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:32.001632 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c9c90e4-15c2-4086-bc40-a99a94fe862c" path="/var/lib/kubelet/pods/0c9c90e4-15c2-4086-bc40-a99a94fe862c/volumes" Jul 08 16:11:32.183213 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:32.183170 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" event={"ID":"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7","Type":"ContainerStarted","Data":"7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4"} Jul 08 16:11:32.183393 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:32.183229 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" event={"ID":"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7","Type":"ContainerStarted","Data":"668a27bfbcce0f6a5d7257c6d494075482e58e4077dd0337f26aa0c5f1e713b4"} Jul 08 16:11:32.183393 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:32.183303 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:32.209146 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:32.209098 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" podStartSLOduration=1.209053791 podStartE2EDuration="1.209053791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:11:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:11:32.208395691 +0000 UTC m=+494.755240787" watchObservedRunningTime="2026-07-08 16:11:32.209053791 +0000 UTC m=+494.755898905" Jul 08 16:11:43.189690 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:43.189655 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:11:50.204806 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.204775 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:11:50.205331 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.205308 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c9c90e4-15c2-4086-bc40-a99a94fe862c" containerName="manager" Jul 08 16:11:50.205403 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.205334 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c9c90e4-15c2-4086-bc40-a99a94fe862c" containerName="manager" Jul 08 16:11:50.205452 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.205421 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c9c90e4-15c2-4086-bc40-a99a94fe862c" containerName="manager" Jul 08 16:11:50.208038 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.208018 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:50.211110 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.211088 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 08 16:11:50.211226 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.211170 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-hbtxw\"" Jul 08 16:11:50.222232 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.222207 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:11:50.257294 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.257269 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e28216f-f511-43c6-a7cf-3c7d2ba77f6e-config-file\") pod \"limitador-limitador-54d6d988d5-7xjhr\" (UID: \"6e28216f-f511-43c6-a7cf-3c7d2ba77f6e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:50.257398 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.257305 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76rff\" (UniqueName: \"kubernetes.io/projected/6e28216f-f511-43c6-a7cf-3c7d2ba77f6e-kube-api-access-76rff\") pod \"limitador-limitador-54d6d988d5-7xjhr\" (UID: \"6e28216f-f511-43c6-a7cf-3c7d2ba77f6e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:50.257692 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.257674 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:11:50.358330 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.358296 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e28216f-f511-43c6-a7cf-3c7d2ba77f6e-config-file\") pod \"limitador-limitador-54d6d988d5-7xjhr\" (UID: \"6e28216f-f511-43c6-a7cf-3c7d2ba77f6e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:50.358535 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.358344 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76rff\" (UniqueName: \"kubernetes.io/projected/6e28216f-f511-43c6-a7cf-3c7d2ba77f6e-kube-api-access-76rff\") pod \"limitador-limitador-54d6d988d5-7xjhr\" (UID: \"6e28216f-f511-43c6-a7cf-3c7d2ba77f6e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:50.359007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.358974 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/6e28216f-f511-43c6-a7cf-3c7d2ba77f6e-config-file\") pod \"limitador-limitador-54d6d988d5-7xjhr\" (UID: \"6e28216f-f511-43c6-a7cf-3c7d2ba77f6e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:50.367600 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.367570 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76rff\" (UniqueName: \"kubernetes.io/projected/6e28216f-f511-43c6-a7cf-3c7d2ba77f6e-kube-api-access-76rff\") pod \"limitador-limitador-54d6d988d5-7xjhr\" (UID: \"6e28216f-f511-43c6-a7cf-3c7d2ba77f6e\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:50.517756 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.517670 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:50.649453 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:50.649397 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:11:50.652291 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:11:50.652264 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e28216f_f511_43c6_a7cf_3c7d2ba77f6e.slice/crio-45ab5311ce7c61563506dc310914e8278dbdb08fe36b7420170ba8c555988ea8 WatchSource:0}: Error finding container 45ab5311ce7c61563506dc310914e8278dbdb08fe36b7420170ba8c555988ea8: Status 404 returned error can't find the container with id 45ab5311ce7c61563506dc310914e8278dbdb08fe36b7420170ba8c555988ea8 Jul 08 16:11:51.271307 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:51.271244 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" event={"ID":"6e28216f-f511-43c6-a7cf-3c7d2ba77f6e","Type":"ContainerStarted","Data":"45ab5311ce7c61563506dc310914e8278dbdb08fe36b7420170ba8c555988ea8"} Jul 08 16:11:53.284500 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:53.284458 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" event={"ID":"6e28216f-f511-43c6-a7cf-3c7d2ba77f6e","Type":"ContainerStarted","Data":"0c404b9852e3336795f9bc7f04312af7c21a355a1c8c9e4e49fcdbc4097f32a8"} Jul 08 16:11:53.284878 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:53.284579 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:11:53.303399 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:11:53.303347 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" podStartSLOduration=0.843052848 podStartE2EDuration="3.303331003s" totalImagesPullingTime="2.460278155s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:11:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:11:53.30278976 +0000 UTC m=+515.849634878" watchObservedRunningTime="2026-07-08 16:11:53.303331003 +0000 UTC m=+515.850176117" Jul 08 16:12:04.290941 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:12:04.290909 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-7xjhr" Jul 08 16:13:04.293297 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.293254 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-5ll5d"] Jul 08 16:13:04.295824 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.295793 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:04.297112 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.297087 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:13:04.298371 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.298352 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pqkmg\"" Jul 08 16:13:04.298371 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.298363 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 08 16:13:04.313683 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.313637 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-5ll5d"] Jul 08 16:13:04.365029 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.364998 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svrr9\" (UniqueName: \"kubernetes.io/projected/b241870a-38a7-4fa6-8549-82be003e21f4-kube-api-access-svrr9\") pod \"payload-pre-processing-7964457696-5ll5d\" (UID: \"b241870a-38a7-4fa6-8549-82be003e21f4\") " pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:04.365236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.365115 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b241870a-38a7-4fa6-8549-82be003e21f4-plugins-config-volume\") pod \"payload-pre-processing-7964457696-5ll5d\" (UID: \"b241870a-38a7-4fa6-8549-82be003e21f4\") " pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:04.466237 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.466199 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svrr9\" (UniqueName: \"kubernetes.io/projected/b241870a-38a7-4fa6-8549-82be003e21f4-kube-api-access-svrr9\") pod \"payload-pre-processing-7964457696-5ll5d\" (UID: \"b241870a-38a7-4fa6-8549-82be003e21f4\") " pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:04.466411 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.466255 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b241870a-38a7-4fa6-8549-82be003e21f4-plugins-config-volume\") pod \"payload-pre-processing-7964457696-5ll5d\" (UID: \"b241870a-38a7-4fa6-8549-82be003e21f4\") " pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:04.467138 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.467115 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b241870a-38a7-4fa6-8549-82be003e21f4-plugins-config-volume\") pod \"payload-pre-processing-7964457696-5ll5d\" (UID: \"b241870a-38a7-4fa6-8549-82be003e21f4\") " pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:04.475673 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.475649 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svrr9\" (UniqueName: \"kubernetes.io/projected/b241870a-38a7-4fa6-8549-82be003e21f4-kube-api-access-svrr9\") pod \"payload-pre-processing-7964457696-5ll5d\" (UID: \"b241870a-38a7-4fa6-8549-82be003e21f4\") " pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:04.606128 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.606023 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:04.752585 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:04.752560 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7964457696-5ll5d"] Jul 08 16:13:04.755266 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:13:04.755235 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb241870a_38a7_4fa6_8549_82be003e21f4.slice/crio-285a07fd4d8ebce2ec09295d4ab5f3108b5ca7e375cdb6a9d6ae5b35e8481f84 WatchSource:0}: Error finding container 285a07fd4d8ebce2ec09295d4ab5f3108b5ca7e375cdb6a9d6ae5b35e8481f84: Status 404 returned error can't find the container with id 285a07fd4d8ebce2ec09295d4ab5f3108b5ca7e375cdb6a9d6ae5b35e8481f84 Jul 08 16:13:05.604609 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:05.604574 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" event={"ID":"b241870a-38a7-4fa6-8549-82be003e21f4","Type":"ContainerStarted","Data":"285a07fd4d8ebce2ec09295d4ab5f3108b5ca7e375cdb6a9d6ae5b35e8481f84"} Jul 08 16:13:06.610549 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:06.610458 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" event={"ID":"b241870a-38a7-4fa6-8549-82be003e21f4","Type":"ContainerStarted","Data":"7cd83f1d49bbd84e642249a246e6536168a213af21ce85e5dfd408ba1f4c6b9b"} Jul 08 16:13:06.610914 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:06.610585 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:06.626704 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:06.626659 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" podStartSLOduration=1.17698014 podStartE2EDuration="2.626646043s" totalImagesPullingTime="1.449665903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:13:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:13:06.626085619 +0000 UTC m=+589.172930725" watchObservedRunningTime="2026-07-08 16:13:06.626646043 +0000 UTC m=+589.173491158" Jul 08 16:13:17.617642 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:17.617612 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7964457696-5ll5d" Jul 08 16:13:17.929907 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:17.929879 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:13:17.933553 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:17.933521 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:13:22.520887 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.520854 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch"] Jul 08 16:13:22.523251 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.523233 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.528275 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.528255 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 08 16:13:22.528411 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.528304 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-t6v4q\"" Jul 08 16:13:22.528411 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.528313 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 08 16:13:22.528411 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.528304 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 08 16:13:22.536351 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.536330 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch"] Jul 08 16:13:22.622909 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.622875 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.623125 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.622921 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.623125 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.622943 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564z6\" (UniqueName: \"kubernetes.io/projected/651ef463-7509-4d65-806f-7cc87bccaaa6-kube-api-access-564z6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.623125 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.622969 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.623125 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.623076 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.623125 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.623116 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651ef463-7509-4d65-806f-7cc87bccaaa6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.723947 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.723914 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.724185 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.723963 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651ef463-7509-4d65-806f-7cc87bccaaa6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.724185 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.724000 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.724185 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.724090 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.724185 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.724121 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564z6\" (UniqueName: \"kubernetes.io/projected/651ef463-7509-4d65-806f-7cc87bccaaa6-kube-api-access-564z6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.724185 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.724147 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.724465 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.724441 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.724525 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.724491 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.724580 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.724542 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.726351 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.726320 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/651ef463-7509-4d65-806f-7cc87bccaaa6-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.726534 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.726520 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/651ef463-7509-4d65-806f-7cc87bccaaa6-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.734750 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.734724 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564z6\" (UniqueName: \"kubernetes.io/projected/651ef463-7509-4d65-806f-7cc87bccaaa6-kube-api-access-564z6\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch\" (UID: \"651ef463-7509-4d65-806f-7cc87bccaaa6\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.834738 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.834663 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:22.842024 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.841994 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:13:22.968458 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:22.968438 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch"] Jul 08 16:13:22.970334 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:13:22.970305 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651ef463_7509_4d65_806f_7cc87bccaaa6.slice/crio-051c112c3de65ff32cd5d00f00cf90850623d7414fe4a53e9c16eb7a82053487 WatchSource:0}: Error finding container 051c112c3de65ff32cd5d00f00cf90850623d7414fe4a53e9c16eb7a82053487: Status 404 returned error can't find the container with id 051c112c3de65ff32cd5d00f00cf90850623d7414fe4a53e9c16eb7a82053487 Jul 08 16:13:23.688753 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:23.688712 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" event={"ID":"651ef463-7509-4d65-806f-7cc87bccaaa6","Type":"ContainerStarted","Data":"051c112c3de65ff32cd5d00f00cf90850623d7414fe4a53e9c16eb7a82053487"} Jul 08 16:13:24.694900 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:24.694866 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" event={"ID":"651ef463-7509-4d65-806f-7cc87bccaaa6","Type":"ContainerStarted","Data":"dc3c506b4a807ce5f202ef998c0f4cc902666b3b06040456531701391d9e1235"} Jul 08 16:13:24.716010 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:24.715952 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" podStartSLOduration=1.727273817 podStartE2EDuration="2.715930776s" totalImagesPullingTime="988.656959ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:13:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:13:24.715420568 +0000 UTC m=+607.262265684" watchObservedRunningTime="2026-07-08 16:13:24.715930776 +0000 UTC m=+607.262775891" Jul 08 16:13:32.835131 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:32.835089 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:32.835689 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:32.835154 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:32.847890 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:32.847863 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:33.745149 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:33.745119 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch" Jul 08 16:13:38.016432 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:38.016400 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:13:42.419783 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:42.419703 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:13:51.501591 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.501554 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv"] Jul 08 16:13:51.504135 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.504115 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.507054 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.507034 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Jul 08 16:13:51.521016 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.520996 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv"] Jul 08 16:13:51.558137 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.558106 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.558282 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.558161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djg6w\" (UniqueName: \"kubernetes.io/projected/4f754de9-d72e-4387-b33b-34168c7cd116-kube-api-access-djg6w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.558282 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.558189 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.558282 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.558208 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.558282 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.558235 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f754de9-d72e-4387-b33b-34168c7cd116-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.558414 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.558297 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.658785 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.658747 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f754de9-d72e-4387-b33b-34168c7cd116-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.658785 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.658792 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.658989 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.658816 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.659039 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.658988 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djg6w\" (UniqueName: \"kubernetes.io/projected/4f754de9-d72e-4387-b33b-34168c7cd116-kube-api-access-djg6w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.659112 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.659093 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.659172 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.659130 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.659225 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.659206 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.659295 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.659269 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.659593 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.659572 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.661254 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.661232 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/4f754de9-d72e-4387-b33b-34168c7cd116-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.661494 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.661474 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/4f754de9-d72e-4387-b33b-34168c7cd116-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.667642 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.667621 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djg6w\" (UniqueName: \"kubernetes.io/projected/4f754de9-d72e-4387-b33b-34168c7cd116-kube-api-access-djg6w\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv\" (UID: \"4f754de9-d72e-4387-b33b-34168c7cd116\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.814257 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.814166 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:13:51.945658 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.945628 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv"] Jul 08 16:13:51.947133 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:13:51.947107 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f754de9_d72e_4387_b33b_34168c7cd116.slice/crio-ebb21c57e43fbe14334a8e05d2250442d688a19133a088657122e535085475a0 WatchSource:0}: Error finding container ebb21c57e43fbe14334a8e05d2250442d688a19133a088657122e535085475a0: Status 404 returned error can't find the container with id ebb21c57e43fbe14334a8e05d2250442d688a19133a088657122e535085475a0 Jul 08 16:13:51.948532 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:51.948509 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 16:13:52.819447 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:52.819413 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" event={"ID":"4f754de9-d72e-4387-b33b-34168c7cd116","Type":"ContainerStarted","Data":"97dd7075e842e31aa2894de33e9439e0da51007d3e42f497081ba67a957d9d90"} Jul 08 16:13:52.819810 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:52.819477 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" event={"ID":"4f754de9-d72e-4387-b33b-34168c7cd116","Type":"ContainerStarted","Data":"ebb21c57e43fbe14334a8e05d2250442d688a19133a088657122e535085475a0"} Jul 08 16:13:52.836910 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:52.836859 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" podStartSLOduration=1.6389719409999999 podStartE2EDuration="1.836844675s" totalImagesPullingTime="197.872734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:13:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:13:52.83568719 +0000 UTC m=+635.382532303" watchObservedRunningTime="2026-07-08 16:13:52.836844675 +0000 UTC m=+635.383689790" Jul 08 16:13:53.333757 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:13:53.333715 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:14:01.815216 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:01.815173 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:14:01.815594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:01.815241 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:14:01.828013 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:01.827988 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:14:01.872279 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:01.872248 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv" Jul 08 16:14:02.112415 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:02.112322 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:14:03.889974 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.889941 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw"] Jul 08 16:14:03.893944 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.893929 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:03.896628 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.896610 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\"" Jul 08 16:14:03.906258 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.906239 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw"] Jul 08 16:14:03.968127 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.968089 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:03.968310 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.968137 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:03.968310 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.968161 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89de82cb-7bf0-4ab2-94f4-a636eac46462-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:03.968310 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.968214 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:03.968310 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.968243 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:03.968310 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:03.968273 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m94g\" (UniqueName: \"kubernetes.io/projected/89de82cb-7bf0-4ab2-94f4-a636eac46462-kube-api-access-9m94g\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.069606 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.069569 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.069788 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.069626 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.069788 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.069664 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m94g\" (UniqueName: \"kubernetes.io/projected/89de82cb-7bf0-4ab2-94f4-a636eac46462-kube-api-access-9m94g\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.069788 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.069717 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.069788 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.069741 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.069788 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.069767 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89de82cb-7bf0-4ab2-94f4-a636eac46462-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.070096 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.070049 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.070177 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.070103 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.070177 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.070167 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.072217 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.072187 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/89de82cb-7bf0-4ab2-94f4-a636eac46462-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.072418 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.072401 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/89de82cb-7bf0-4ab2-94f4-a636eac46462-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.078081 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.078040 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m94g\" (UniqueName: \"kubernetes.io/projected/89de82cb-7bf0-4ab2-94f4-a636eac46462-kube-api-access-9m94g\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw\" (UID: \"89de82cb-7bf0-4ab2-94f4-a636eac46462\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.203250 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.203216 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:04.336047 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.335992 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw"] Jul 08 16:14:04.338381 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:14:04.338353 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod89de82cb_7bf0_4ab2_94f4_a636eac46462.slice/crio-826b5e8b1d7051abce5fe6e8138f6725959c1f53d7df29d11e2ac68edc303271 WatchSource:0}: Error finding container 826b5e8b1d7051abce5fe6e8138f6725959c1f53d7df29d11e2ac68edc303271: Status 404 returned error can't find the container with id 826b5e8b1d7051abce5fe6e8138f6725959c1f53d7df29d11e2ac68edc303271 Jul 08 16:14:04.875996 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.875911 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" event={"ID":"89de82cb-7bf0-4ab2-94f4-a636eac46462","Type":"ContainerStarted","Data":"0f1abc8afe5683a778359e5a229b6dc66451755069568eaaedb57a6440f1f794"} Jul 08 16:14:04.875996 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.875961 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" event={"ID":"89de82cb-7bf0-4ab2-94f4-a636eac46462","Type":"ContainerStarted","Data":"826b5e8b1d7051abce5fe6e8138f6725959c1f53d7df29d11e2ac68edc303271"} Jul 08 16:14:04.896388 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:04.896335 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" podStartSLOduration=1.6800852210000001 podStartE2EDuration="1.896319671s" totalImagesPullingTime="216.23445ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:14:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:14:04.895995781 +0000 UTC m=+647.442840896" watchObservedRunningTime="2026-07-08 16:14:04.896319671 +0000 UTC m=+647.443164784" Jul 08 16:14:12.221599 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:12.221564 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:14:14.204132 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:14.204095 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:14.204555 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:14.204146 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:14.216526 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:14.216498 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:14:14.931734 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:14:14.931702 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw" Jul 08 16:15:44.524029 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:15:44.523994 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:15:53.413078 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:15:53.413040 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:16:03.712865 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:16:03.712831 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:16:13.615128 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:16:13.615089 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:16:24.320787 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:16:24.320752 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:16:50.520127 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:16:50.520027 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:17:01.736228 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:17:01.736195 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:17:38.217373 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:17:38.217339 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:17:47.617394 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:17:47.617352 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:17:56.614590 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:17:56.614550 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:18:02.517957 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:18:02.517915 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:18:12.824231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:18:12.824151 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:18:17.960185 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:18:17.960154 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:18:17.962396 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:18:17.962370 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:18:23.313891 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:18:23.313856 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:18:34.419383 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:18:34.419344 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:18:43.811689 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:18:43.811646 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:19:06.524440 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:19:06.524390 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:19:17.910986 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:19:17.910947 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:19:42.313985 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:19:42.313910 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:20:00.973297 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:20:00.973259 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:20:11.609406 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:20:11.609373 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:20:23.114142 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:20:23.114108 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:20:32.118474 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:20:32.118439 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:20:41.411294 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:20:41.411255 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:20:50.411895 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:20:50.411855 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:21:16.666372 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:16.666279 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk"] Jul 08 16:21:16.666925 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:16.666509 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" podUID="aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7" containerName="manager" containerID="cri-o://7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4" gracePeriod=10 Jul 08 16:21:18.714702 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.714677 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:21:18.822455 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.822378 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cfmls\" (UniqueName: \"kubernetes.io/projected/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-kube-api-access-cfmls\") pod \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\" (UID: \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\") " Jul 08 16:21:18.822589 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.822463 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-extensions-socket-volume\") pod \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\" (UID: \"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7\") " Jul 08 16:21:18.822847 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.822823 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7" (UID: "aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:21:18.824421 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.824396 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-kube-api-access-cfmls" (OuterVolumeSpecName: "kube-api-access-cfmls") pod "aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7" (UID: "aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7"). InnerVolumeSpecName "kube-api-access-cfmls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:21:18.887053 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.887017 2563 generic.go:356] "Generic (PLEG): container finished" podID="aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7" containerID="7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4" exitCode=0 Jul 08 16:21:18.887243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.887079 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" Jul 08 16:21:18.887243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.887110 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" event={"ID":"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7","Type":"ContainerDied","Data":"7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4"} Jul 08 16:21:18.887243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.887158 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk" event={"ID":"aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7","Type":"ContainerDied","Data":"668a27bfbcce0f6a5d7257c6d494075482e58e4077dd0337f26aa0c5f1e713b4"} Jul 08 16:21:18.887243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.887180 2563 scope.go:117] "RemoveContainer" containerID="7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4" Jul 08 16:21:18.896860 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.896844 2563 scope.go:117] "RemoveContainer" containerID="7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4" Jul 08 16:21:18.897125 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:21:18.897107 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4\": container with ID starting with 7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4 not found: ID does not exist" containerID="7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4" Jul 08 16:21:18.897179 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.897133 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4"} err="failed to get container status \"7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4\": rpc error: code = NotFound desc = could not find container \"7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4\": container with ID starting with 7f1567e9e3c3b4c63ef3236ce12c3e797ebda64d05d96d0711d0b4c5d47d26a4 not found: ID does not exist" Jul 08 16:21:18.909544 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.909517 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk"] Jul 08 16:21:18.912748 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.912730 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-7k9jk"] Jul 08 16:21:18.923420 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.923402 2563 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-extensions-socket-volume\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:21:18.923487 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:18.923423 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfmls\" (UniqueName: \"kubernetes.io/projected/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7-kube-api-access-cfmls\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:21:20.001939 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:21:20.001902 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7" path="/var/lib/kubelet/pods/aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7/volumes" Jul 08 16:22:17.844956 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.844915 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p"] Jul 08 16:22:17.845502 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.845472 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7" containerName="manager" Jul 08 16:22:17.845502 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.845492 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7" containerName="manager" Jul 08 16:22:17.845614 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.845606 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="aed6c1ce-dcfb-46a9-968b-7009c4d3a4a7" containerName="manager" Jul 08 16:22:17.848657 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.848637 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:17.851527 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.851506 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hdf7b\"" Jul 08 16:22:17.863325 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.863298 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p"] Jul 08 16:22:17.934881 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.934850 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5wcz\" (UniqueName: \"kubernetes.io/projected/86c469d2-ab9f-4e5c-ab8a-9666512275a8-kube-api-access-f5wcz\") pod \"kuadrant-operator-controller-manager-86878c769f-jkw5p\" (UID: \"86c469d2-ab9f-4e5c-ab8a-9666512275a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:17.935042 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:17.934921 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86c469d2-ab9f-4e5c-ab8a-9666512275a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jkw5p\" (UID: \"86c469d2-ab9f-4e5c-ab8a-9666512275a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:18.035633 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:18.035605 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5wcz\" (UniqueName: \"kubernetes.io/projected/86c469d2-ab9f-4e5c-ab8a-9666512275a8-kube-api-access-f5wcz\") pod \"kuadrant-operator-controller-manager-86878c769f-jkw5p\" (UID: \"86c469d2-ab9f-4e5c-ab8a-9666512275a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:18.035709 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:18.035672 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86c469d2-ab9f-4e5c-ab8a-9666512275a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jkw5p\" (UID: \"86c469d2-ab9f-4e5c-ab8a-9666512275a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:18.035972 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:18.035956 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/86c469d2-ab9f-4e5c-ab8a-9666512275a8-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-jkw5p\" (UID: \"86c469d2-ab9f-4e5c-ab8a-9666512275a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:18.046233 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:18.046206 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5wcz\" (UniqueName: \"kubernetes.io/projected/86c469d2-ab9f-4e5c-ab8a-9666512275a8-kube-api-access-f5wcz\") pod \"kuadrant-operator-controller-manager-86878c769f-jkw5p\" (UID: \"86c469d2-ab9f-4e5c-ab8a-9666512275a8\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:18.162075 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:18.162047 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-hdf7b\"" Jul 08 16:22:18.169888 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:18.169869 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:18.311875 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:18.311842 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p"] Jul 08 16:22:18.314989 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:22:18.314962 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86c469d2_ab9f_4e5c_ab8a_9666512275a8.slice/crio-6018c277ae9bb16cd36830a9e7a445d595f4bb0cadce97b898c1a6dd8e8969ce WatchSource:0}: Error finding container 6018c277ae9bb16cd36830a9e7a445d595f4bb0cadce97b898c1a6dd8e8969ce: Status 404 returned error can't find the container with id 6018c277ae9bb16cd36830a9e7a445d595f4bb0cadce97b898c1a6dd8e8969ce Jul 08 16:22:19.163828 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:19.163798 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" event={"ID":"86c469d2-ab9f-4e5c-ab8a-9666512275a8","Type":"ContainerStarted","Data":"22423cfb2525945214156ce5dc75397bc595a374a94650e3e2d5372d63fd9c3e"} Jul 08 16:22:19.164268 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:19.163849 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" event={"ID":"86c469d2-ab9f-4e5c-ab8a-9666512275a8","Type":"ContainerStarted","Data":"6018c277ae9bb16cd36830a9e7a445d595f4bb0cadce97b898c1a6dd8e8969ce"} Jul 08 16:22:19.164268 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:19.163952 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:19.188250 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:19.188203 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" podStartSLOduration=2.188188941 podStartE2EDuration="2.188188941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:22:19.185766179 +0000 UTC m=+1141.732611307" watchObservedRunningTime="2026-07-08 16:22:19.188188941 +0000 UTC m=+1141.735034054" Jul 08 16:22:30.171175 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:30.171144 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-jkw5p" Jul 08 16:22:39.712383 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:39.712343 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:22:49.310781 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:22:49.310701 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:23:02.222129 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:23:02.222092 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:23:12.417971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:23:12.417935 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:23:17.991325 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:23:17.991296 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:23:17.991994 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:23:17.991974 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:23:24.415038 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:23:24.415005 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:23:34.624560 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:23:34.624524 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:23:43.723739 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:23:43.723695 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:23:53.912911 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:23:53.912878 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:24:04.418837 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:24:04.418800 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:24:14.416189 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:24:14.416103 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:24:26.019971 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:24:26.019936 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:24:51.416710 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:24:51.416675 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:24:59.018444 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:24:59.018409 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:25:08.610126 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:25:08.610087 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:25:19.415486 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:25:19.415451 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:25:27.422258 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:25:27.422224 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:25:39.113079 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:25:39.113034 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:25:50.616359 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:25:50.616282 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:26:00.011166 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:26:00.011134 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:26:07.912862 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:26:07.912826 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:26:18.716191 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:26:18.716159 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:26:28.413823 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:26:28.413789 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:26:41.320289 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:26:41.320253 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:26:50.817835 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:26:50.817801 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:27:00.020086 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:27:00.020028 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:27:11.818816 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:27:11.818723 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:27:23.715146 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:27:23.715107 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:27:30.523981 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:27:30.523940 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:27:47.220460 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:27:47.220419 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:27:59.325388 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:27:59.325355 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:28:08.525330 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:08.525291 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:28:18.018614 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:18.018583 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:28:18.019036 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:18.019020 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:28:22.248478 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:22.248433 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:28:31.329168 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:31.329137 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:28:43.441983 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:43.441949 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:28:57.220622 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.220588 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f"] Jul 08 16:28:57.224418 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.224396 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.232632 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.232607 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-bb93692f-openshift-default-dockercfg-bldzc\"" Jul 08 16:28:57.233303 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.232908 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 16:28:57.265373 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.265343 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f"] Jul 08 16:28:57.286305 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286275 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-podinfo\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.286439 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286324 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-data\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.286439 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286361 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-certs\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.286439 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286391 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-credential-socket\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.286439 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286415 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-envoy\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.286439 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286430 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-token\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.286617 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286446 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-socket\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.286617 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286472 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8szf\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-kube-api-access-n8szf\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.286617 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.286500 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e911cdf-5746-408c-95f6-6a10305e03d1-istiod-ca-cert\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.387714 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387680 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-data\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.387885 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387732 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-certs\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.387885 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387760 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-credential-socket\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.387885 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387779 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-envoy\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.387885 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387793 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-token\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.387885 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387808 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-socket\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.387885 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387823 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8szf\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-kube-api-access-n8szf\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.387885 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387862 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e911cdf-5746-408c-95f6-6a10305e03d1-istiod-ca-cert\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.388236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.387923 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-podinfo\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.388236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.388156 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-data\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.388236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.388170 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-credential-socket\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.388399 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.388374 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-certs\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.388452 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.388375 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-socket\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.388783 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.388761 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e911cdf-5746-408c-95f6-6a10305e03d1-istiod-ca-cert\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.390123 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.390100 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-envoy\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.390505 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.390489 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-podinfo\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.412387 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.412358 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8szf\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-kube-api-access-n8szf\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.413351 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.413327 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-token\") pod \"e2e-ait-bb93692f-openshift-default-699d957665-pz69f\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.537970 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.537889 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:28:57.693500 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.693351 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f"] Jul 08 16:28:57.696214 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:28:57.696189 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e911cdf_5746_408c_95f6_6a10305e03d1.slice/crio-8fb03c4b2364ac849a556e64fab5f3cb62851ab9a77f6a56ea2fb9a77c55996b WatchSource:0}: Error finding container 8fb03c4b2364ac849a556e64fab5f3cb62851ab9a77f6a56ea2fb9a77c55996b: Status 404 returned error can't find the container with id 8fb03c4b2364ac849a556e64fab5f3cb62851ab9a77f6a56ea2fb9a77c55996b Jul 08 16:28:57.698236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:57.698216 2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 08 16:28:58.029469 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:28:58.029424 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" event={"ID":"5e911cdf-5746-408c-95f6-6a10305e03d1","Type":"ContainerStarted","Data":"8fb03c4b2364ac849a556e64fab5f3cb62851ab9a77f6a56ea2fb9a77c55996b"} Jul 08 16:29:03.127685 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:03.127650 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:29:09.882648 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:09.882610 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f"] Jul 08 16:29:12.066725 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:12.066634 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:29:24.366435 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:24.366395 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:29:24.366737 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:24.366488 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:29:24.366737 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:24.366520 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:29:25.157852 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:25.157797 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" event={"ID":"5e911cdf-5746-408c-95f6-6a10305e03d1","Type":"ContainerStarted","Data":"e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee"} Jul 08 16:29:25.158095 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:25.157857 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" podUID="5e911cdf-5746-408c-95f6-6a10305e03d1" containerName="istio-proxy" containerID="cri-o://e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee" gracePeriod=30 Jul 08 16:29:25.181472 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:25.181421 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" podStartSLOduration=1.513664758 podStartE2EDuration="28.181405257s" totalImagesPullingTime="26.667740499s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:28:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:29:25.18030677 +0000 UTC m=+1567.727151890" watchObservedRunningTime="2026-07-08 16:29:25.181405257 +0000 UTC m=+1567.728250427" Jul 08 16:29:27.539026 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:27.538986 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:29:30.421169 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.421109 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:29:30.591516 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591486 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-data\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.591722 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591545 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-credential-socket\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.591722 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591564 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-certs\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.591722 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591580 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-socket\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.591722 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591615 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-token\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.591722 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591653 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8szf\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-kube-api-access-n8szf\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.591722 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591698 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-envoy\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.591722 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591724 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-podinfo\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.592119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591765 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e911cdf-5746-408c-95f6-6a10305e03d1-istiod-ca-cert\") pod \"5e911cdf-5746-408c-95f6-6a10305e03d1\" (UID: \"5e911cdf-5746-408c-95f6-6a10305e03d1\") " Jul 08 16:29:30.592119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591820 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-data" (OuterVolumeSpecName: "istio-data") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:29:30.592119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591957 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:29:30.592119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591971 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:29:30.592119 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.591922 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:29:30.592375 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.592128 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-data\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:30.592375 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.592148 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-credential-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:30.592375 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.592163 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-certs\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:30.592375 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.592177 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-workload-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:30.592582 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.592429 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e911cdf-5746-408c-95f6-6a10305e03d1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 16:29:30.594163 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.594132 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-kube-api-access-n8szf" (OuterVolumeSpecName: "kube-api-access-n8szf") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "kube-api-access-n8szf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:29:30.594290 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.594206 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-token" (OuterVolumeSpecName: "istio-token") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:29:30.594412 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.594391 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:29:30.594449 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.594398 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5e911cdf-5746-408c-95f6-6a10305e03d1" (UID: "5e911cdf-5746-408c-95f6-6a10305e03d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 16:29:30.692864 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.692777 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-token\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:30.692864 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.692809 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8szf\" (UniqueName: \"kubernetes.io/projected/5e911cdf-5746-408c-95f6-6a10305e03d1-kube-api-access-n8szf\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:30.692864 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.692820 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-envoy\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:30.692864 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.692828 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e911cdf-5746-408c-95f6-6a10305e03d1-istio-podinfo\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:30.692864 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:30.692837 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e911cdf-5746-408c-95f6-6a10305e03d1-istiod-ca-cert\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:29:31.185892 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.185852 2563 generic.go:356] "Generic (PLEG): container finished" podID="5e911cdf-5746-408c-95f6-6a10305e03d1" containerID="e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee" exitCode=0 Jul 08 16:29:31.186079 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.185899 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" Jul 08 16:29:31.186079 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.185932 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" event={"ID":"5e911cdf-5746-408c-95f6-6a10305e03d1","Type":"ContainerDied","Data":"e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee"} Jul 08 16:29:31.186079 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.185985 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f" event={"ID":"5e911cdf-5746-408c-95f6-6a10305e03d1","Type":"ContainerDied","Data":"8fb03c4b2364ac849a556e64fab5f3cb62851ab9a77f6a56ea2fb9a77c55996b"} Jul 08 16:29:31.186079 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.186007 2563 scope.go:117] "RemoveContainer" containerID="e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee" Jul 08 16:29:31.195120 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.195101 2563 scope.go:117] "RemoveContainer" containerID="e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee" Jul 08 16:29:31.195387 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:29:31.195369 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee\": container with ID starting with e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee not found: ID does not exist" containerID="e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee" Jul 08 16:29:31.195440 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.195395 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee"} err="failed to get container status \"e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee\": rpc error: code = NotFound desc = could not find container \"e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee\": container with ID starting with e7f9ceb5c5ed5846af0a212bbee734c68f47f80ad7b6bd621d02777592cf1aee not found: ID does not exist" Jul 08 16:29:31.209223 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.209198 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f"] Jul 08 16:29:31.212948 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:31.212925 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-bb93692f-openshift-default-699d957665-pz69f"] Jul 08 16:29:32.001875 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:29:32.001842 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e911cdf-5746-408c-95f6-6a10305e03d1" path="/var/lib/kubelet/pods/5e911cdf-5746-408c-95f6-6a10305e03d1/volumes" Jul 08 16:30:53.116296 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:30:53.116261 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:31:02.018874 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:02.018842 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:31:05.823965 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.823932 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj"] Jul 08 16:31:05.824562 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.824541 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e911cdf-5746-408c-95f6-6a10305e03d1" containerName="istio-proxy" Jul 08 16:31:05.824687 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.824565 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e911cdf-5746-408c-95f6-6a10305e03d1" containerName="istio-proxy" Jul 08 16:31:05.824687 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.824658 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e911cdf-5746-408c-95f6-6a10305e03d1" containerName="istio-proxy" Jul 08 16:31:05.827901 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.827878 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.830654 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.830632 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b2584c54-openshift-default-dockercfg-q9klj\"" Jul 08 16:31:05.830858 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.830844 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 16:31:05.839758 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.839733 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj"] Jul 08 16:31:05.881680 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881648 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-envoy\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.881680 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881689 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-credential-socket\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.881909 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881717 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8k5pc\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-kube-api-access-8k5pc\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.881909 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881804 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-certs\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.881909 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881833 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-data\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.881909 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881854 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-socket\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.882095 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881924 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968f612-5f51-4bfb-be39-9aad438f55f1-istiod-ca-cert\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.882095 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881952 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-podinfo\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.882095 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.881992 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-token\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.982953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.982915 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-socket\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.982953 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.982958 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968f612-5f51-4bfb-be39-9aad438f55f1-istiod-ca-cert\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.982977 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-podinfo\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983013 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-token\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983053 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-envoy\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983107 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-credential-socket\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983132 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8k5pc\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-kube-api-access-8k5pc\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983174 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-certs\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983243 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983196 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-data\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983584 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983417 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-socket\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983584 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983493 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-credential-socket\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983707 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983688 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-certs\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983765 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983741 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968f612-5f51-4bfb-be39-9aad438f55f1-istiod-ca-cert\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.983823 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.983760 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-data\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.985448 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.985423 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-envoy\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.985546 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.985527 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-podinfo\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.991330 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.991300 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8k5pc\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-kube-api-access-8k5pc\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:05.991442 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:05.991423 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-token\") pod \"e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:06.138272 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:06.138191 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:06.278728 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:06.278662 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj"] Jul 08 16:31:06.281260 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:31:06.281231 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0968f612_5f51_4bfb_be39_9aad438f55f1.slice/crio-2ad6c21d6f39d0d73a0dda2415e4be8210916a2f7fe99663d0412541262e4eb9 WatchSource:0}: Error finding container 2ad6c21d6f39d0d73a0dda2415e4be8210916a2f7fe99663d0412541262e4eb9: Status 404 returned error can't find the container with id 2ad6c21d6f39d0d73a0dda2415e4be8210916a2f7fe99663d0412541262e4eb9 Jul 08 16:31:06.283797 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:06.283767 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:31:06.283881 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:06.283833 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:31:06.283881 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:06.283859 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:31:06.627982 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:06.627942 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" event={"ID":"0968f612-5f51-4bfb-be39-9aad438f55f1","Type":"ContainerStarted","Data":"c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41"} Jul 08 16:31:06.628182 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:06.628004 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" event={"ID":"0968f612-5f51-4bfb-be39-9aad438f55f1","Type":"ContainerStarted","Data":"2ad6c21d6f39d0d73a0dda2415e4be8210916a2f7fe99663d0412541262e4eb9"} Jul 08 16:31:06.650695 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:06.650639 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" podStartSLOduration=1.650618586 podStartE2EDuration="1.650618586s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:31:06.648196275 +0000 UTC m=+1669.195041388" watchObservedRunningTime="2026-07-08 16:31:06.650618586 +0000 UTC m=+1669.197463701" Jul 08 16:31:07.138360 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:07.138332 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:07.143304 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:07.143282 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:07.633861 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:07.633829 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:07.634698 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:07.634680 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:11.224796 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:11.224759 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:31:19.701572 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:19.701539 2563 generic.go:356] "Generic (PLEG): container finished" podID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerID="c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41" exitCode=0 Jul 08 16:31:19.702145 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:19.701617 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" event={"ID":"0968f612-5f51-4bfb-be39-9aad438f55f1","Type":"ContainerDied","Data":"c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41"} Jul 08 16:31:19.702222 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:19.702142 2563 scope.go:117] "RemoveContainer" containerID="c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41" Jul 08 16:31:19.703226 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:19.703193 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:31:19.703339 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:19.703258 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:31:19.703339 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:19.703290 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:31:20.708557 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:20.708524 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" event={"ID":"0968f612-5f51-4bfb-be39-9aad438f55f1","Type":"ContainerStarted","Data":"5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9"} Jul 08 16:31:21.139100 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:21.138998 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:21.140114 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:21.140086 2563 patch_prober.go:28] interesting pod/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 08 16:31:21.140213 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:21.140133 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 08 16:31:22.139383 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:22.139345 2563 patch_prober.go:28] interesting pod/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 08 16:31:22.139784 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:22.139398 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 08 16:31:22.634555 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:22.634524 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:23.139126 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:23.139055 2563 patch_prober.go:28] interesting pod/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 08 16:31:23.139357 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:23.139145 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 08 16:31:24.138980 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:24.138937 2563 patch_prober.go:28] interesting pod/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" start-of-body= Jul 08 16:31:24.139489 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:24.138993 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.39:15021/healthz/ready\": dial tcp 10.132.0.39:15021: connect: connection refused" Jul 08 16:31:24.612153 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:24.612119 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:31:25.142624 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:25.142589 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:25.143611 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:25.143593 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:27.469009 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:27.468979 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj"] Jul 08 16:31:27.469473 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:27.469280 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" containerID="cri-o://5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9" gracePeriod=30 Jul 08 16:31:32.733909 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.733842 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:32.768050 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.768018 2563 generic.go:356] "Generic (PLEG): container finished" podID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerID="5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9" exitCode=0 Jul 08 16:31:32.768228 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.768099 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" Jul 08 16:31:32.768228 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.768098 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" event={"ID":"0968f612-5f51-4bfb-be39-9aad438f55f1","Type":"ContainerDied","Data":"5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9"} Jul 08 16:31:32.768228 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.768159 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj" event={"ID":"0968f612-5f51-4bfb-be39-9aad438f55f1","Type":"ContainerDied","Data":"2ad6c21d6f39d0d73a0dda2415e4be8210916a2f7fe99663d0412541262e4eb9"} Jul 08 16:31:32.768228 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.768190 2563 scope.go:117] "RemoveContainer" containerID="5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9" Jul 08 16:31:32.781273 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.781252 2563 scope.go:117] "RemoveContainer" containerID="c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41" Jul 08 16:31:32.789601 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.789580 2563 scope.go:117] "RemoveContainer" containerID="5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9" Jul 08 16:31:32.789877 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:31:32.789858 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9\": container with ID starting with 5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9 not found: ID does not exist" containerID="5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9" Jul 08 16:31:32.789938 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.789893 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9"} err="failed to get container status \"5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9\": rpc error: code = NotFound desc = could not find container \"5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9\": container with ID starting with 5718a4a7c2ad588cbeb770e35c3f40622dda6b6a8c84863de1cc6d510d9f33f9 not found: ID does not exist" Jul 08 16:31:32.789938 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.789921 2563 scope.go:117] "RemoveContainer" containerID="c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41" Jul 08 16:31:32.790218 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:31:32.790197 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41\": container with ID starting with c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41 not found: ID does not exist" containerID="c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41" Jul 08 16:31:32.790287 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.790222 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41"} err="failed to get container status \"c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41\": rpc error: code = NotFound desc = could not find container \"c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41\": container with ID starting with c6c188efa2d8879bb072ac0526039c0071febdfc79281e96dbfe695ec64bdb41 not found: ID does not exist" Jul 08 16:31:32.835844 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.835810 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-token\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836059 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.835856 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-podinfo\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836059 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.835879 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-envoy\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836059 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.835914 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-data\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836059 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.835942 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-certs\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836059 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.835997 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-socket\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836059 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.836023 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968f612-5f51-4bfb-be39-9aad438f55f1-istiod-ca-cert\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836401 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.836117 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8k5pc\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-kube-api-access-8k5pc\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836401 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.836148 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-credential-socket\") pod \"0968f612-5f51-4bfb-be39-9aad438f55f1\" (UID: \"0968f612-5f51-4bfb-be39-9aad438f55f1\") " Jul 08 16:31:32.836401 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.836322 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:31:32.836558 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.836456 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:32.836558 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.836511 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:31:32.836749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.836716 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0968f612-5f51-4bfb-be39-9aad438f55f1-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 16:31:32.836882 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.836827 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:31:32.837525 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.837497 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-data" (OuterVolumeSpecName: "istio-data") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:31:32.838344 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.838317 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 16:31:32.838451 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.838426 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-kube-api-access-8k5pc" (OuterVolumeSpecName: "kube-api-access-8k5pc") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "kube-api-access-8k5pc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:31:32.838555 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.838540 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-token" (OuterVolumeSpecName: "istio-token") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:31:32.838633 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.838613 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0968f612-5f51-4bfb-be39-9aad438f55f1" (UID: "0968f612-5f51-4bfb-be39-9aad438f55f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:31:32.937656 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.937617 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8k5pc\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-kube-api-access-8k5pc\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:32.937656 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.937649 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-credential-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:32.937656 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.937659 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-token\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:32.937656 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.937667 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-podinfo\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:32.937925 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.937675 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-envoy\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:32.937925 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.937684 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-istio-data\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:32.937925 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.937692 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0968f612-5f51-4bfb-be39-9aad438f55f1-workload-certs\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:32.937925 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:32.937700 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0968f612-5f51-4bfb-be39-9aad438f55f1-istiod-ca-cert\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:31:33.090849 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:33.090752 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj"] Jul 08 16:31:33.095163 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:33.095132 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b2584c54-openshift-default-85d8964f75-hs9xj"] Jul 08 16:31:34.002751 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:34.002719 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" path="/var/lib/kubelet/pods/0968f612-5f51-4bfb-be39-9aad438f55f1/volumes" Jul 08 16:31:56.315980 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:31:56.315899 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:32:30.810111 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:32:30.810078 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:32:50.515655 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:32:50.515620 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:33:05.009236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:33:05.009206 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:33:18.047854 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:33:18.047817 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:33:18.048364 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:33:18.048343 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:33:43.616046 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:33:43.616004 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:33:59.021009 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:33:59.020973 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:34:09.210959 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:09.210920 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:34:52.394317 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.394233 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv"] Jul 08 16:34:52.394768 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.394759 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" Jul 08 16:34:52.394809 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.394774 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" Jul 08 16:34:52.394809 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.394799 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" Jul 08 16:34:52.394809 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.394807 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" Jul 08 16:34:52.394898 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.394889 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" Jul 08 16:34:52.398183 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.398164 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.400853 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.400832 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 16:34:52.400980 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.400875 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-dcd525-openshift-default-dockercfg-qf4tz\"" Jul 08 16:34:52.409216 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.409191 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv"] Jul 08 16:34:52.474316 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474282 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-credential-socket\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.474506 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474330 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-socket\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.474506 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474370 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-podinfo\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.474506 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474439 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-token\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.474506 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474463 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istiod-ca-cert\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.474506 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474491 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-certs\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.474717 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474517 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tskdt\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-kube-api-access-tskdt\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.474717 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474554 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-envoy\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.474717 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.474574 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-data\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.575726 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.575692 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istiod-ca-cert\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.575931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.575745 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-certs\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.575931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.575775 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tskdt\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-kube-api-access-tskdt\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.575931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.575823 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-envoy\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.575931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.575843 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-data\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.575931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.575898 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-credential-socket\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.575931 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.575922 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-socket\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.576266 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.575953 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-podinfo\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.576266 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.576005 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-token\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.576266 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.576167 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-certs\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.576266 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.576251 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-data\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.576482 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.576365 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-socket\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.576597 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.576574 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-credential-socket\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.576775 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.576753 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istiod-ca-cert\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.578509 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.578477 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-envoy\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.578734 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.578713 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-podinfo\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.584747 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.584720 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-token\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.584833 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.584719 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tskdt\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-kube-api-access-tskdt\") pod \"e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.709559 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.709532 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:52.859111 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.859086 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv"] Jul 08 16:34:52.861363 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:34:52.861336 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7213606d_e3d2_4e28_92c4_fdd1508f3c15.slice/crio-d5851eca34051e1adbf3af0aa1b43f806d03db71dcbf478a988fef89f0dc70be WatchSource:0}: Error finding container d5851eca34051e1adbf3af0aa1b43f806d03db71dcbf478a988fef89f0dc70be: Status 404 returned error can't find the container with id d5851eca34051e1adbf3af0aa1b43f806d03db71dcbf478a988fef89f0dc70be Jul 08 16:34:52.863503 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.863469 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:34:52.863610 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.863537 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:34:52.863610 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:52.863567 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:34:53.705116 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:53.705083 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" event={"ID":"7213606d-e3d2-4e28-92c4-fdd1508f3c15","Type":"ContainerStarted","Data":"62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410"} Jul 08 16:34:53.705495 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:53.705136 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" event={"ID":"7213606d-e3d2-4e28-92c4-fdd1508f3c15","Type":"ContainerStarted","Data":"d5851eca34051e1adbf3af0aa1b43f806d03db71dcbf478a988fef89f0dc70be"} Jul 08 16:34:53.709907 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:53.709887 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:53.714523 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:53.714497 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:53.726333 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:53.726294 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" podStartSLOduration=1.726278945 podStartE2EDuration="1.726278945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:34:53.72530494 +0000 UTC m=+1896.272150056" watchObservedRunningTime="2026-07-08 16:34:53.726278945 +0000 UTC m=+1896.273124058" Jul 08 16:34:54.709562 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:54.709529 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:54.710592 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:54.710573 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:34:54.909175 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:34:54.909134 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:35:06.933544 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:06.933507 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:35:09.710830 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:09.710791 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 08 16:35:09.711286 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:09.710883 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 08 16:35:09.789799 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:09.789767 2563 generic.go:356] "Generic (PLEG): container finished" podID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerID="62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410" exitCode=0 Jul 08 16:35:09.789958 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:09.789829 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" event={"ID":"7213606d-e3d2-4e28-92c4-fdd1508f3c15","Type":"ContainerDied","Data":"62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410"} Jul 08 16:35:09.790221 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:09.790207 2563 scope.go:117] "RemoveContainer" containerID="62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410" Jul 08 16:35:09.791273 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:09.791244 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:35:09.791341 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:09.791311 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:35:09.791380 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:09.791344 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:35:10.797228 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:10.797186 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" event={"ID":"7213606d-e3d2-4e28-92c4-fdd1508f3c15","Type":"ContainerStarted","Data":"671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186"} Jul 08 16:35:11.710027 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:11.709993 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:35:11.710223 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:11.710207 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 08 16:35:11.710290 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:11.710270 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 08 16:35:12.710453 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:12.710414 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 08 16:35:12.710856 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:12.710472 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 08 16:35:13.710514 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:13.710477 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body= Jul 08 16:35:13.710908 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:13.710533 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" Jul 08 16:35:14.723746 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:14.723718 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:35:14.724223 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:14.723971 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:35:14.724986 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:14.724967 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:35:16.215564 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:16.215530 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:35:30.008839 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:30.008808 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:35:37.603717 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.603685 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2"] Jul 08 16:35:37.604134 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.604120 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0968f612-5f51-4bfb-be39-9aad438f55f1" containerName="istio-proxy" Jul 08 16:35:37.606144 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.606127 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.609113 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.609087 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dcd525\"/\"openshift-service-ca.crt\"" Jul 08 16:35:37.609239 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.609160 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dcd525\"/\"default-dockercfg-lzwws\"" Jul 08 16:35:37.610439 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.610416 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-dcd525\"/\"test-model-dcd525-kserve-self-signed-certs\"" Jul 08 16:35:37.610574 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.610457 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-dcd525\"/\"kube-root-ca.crt\"" Jul 08 16:35:37.622660 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.622633 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2"] Jul 08 16:35:37.688934 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.688900 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgzm7\" (UniqueName: \"kubernetes.io/projected/d5ea3d53-7068-4df8-87be-71e0e876a17d-kube-api-access-rgzm7\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.688934 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.688936 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-model-cache\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.689184 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.688959 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-tmp-dir\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.689184 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.689111 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-home\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.689184 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.689175 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-dshm\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.689305 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.689203 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ea3d53-7068-4df8-87be-71e0e876a17d-tls-certs\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790045 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790005 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgzm7\" (UniqueName: \"kubernetes.io/projected/d5ea3d53-7068-4df8-87be-71e0e876a17d-kube-api-access-rgzm7\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790045 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-model-cache\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790333 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790059 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-tmp-dir\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790333 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790148 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-home\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790333 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790292 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-dshm\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790471 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790350 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ea3d53-7068-4df8-87be-71e0e876a17d-tls-certs\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790512 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790474 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-model-cache\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790546 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790511 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-home\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.790546 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.790536 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-tmp-dir\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.792595 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.792574 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-dshm\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.792683 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.792648 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ea3d53-7068-4df8-87be-71e0e876a17d-tls-certs\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.799226 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.799204 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgzm7\" (UniqueName: \"kubernetes.io/projected/d5ea3d53-7068-4df8-87be-71e0e876a17d-kube-api-access-rgzm7\") pod \"test-model-dcd525-kserve-699bffff99-v8xs2\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:37.915455 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:37.915413 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:38.059548 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:38.059521 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2"] Jul 08 16:35:38.062285 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:35:38.062251 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd5ea3d53_7068_4df8_87be_71e0e876a17d.slice/crio-93bb5abe21085f54688f363246381f83b91932bb60096cb86b64fc92fab32bae WatchSource:0}: Error finding container 93bb5abe21085f54688f363246381f83b91932bb60096cb86b64fc92fab32bae: Status 404 returned error can't find the container with id 93bb5abe21085f54688f363246381f83b91932bb60096cb86b64fc92fab32bae Jul 08 16:35:38.928317 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:38.928285 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" event={"ID":"d5ea3d53-7068-4df8-87be-71e0e876a17d","Type":"ContainerStarted","Data":"f5cc3261f4e2d9ffc4c8198bfedab02bf2272764b6bd1ba96f9c1e7e6339a823"} Jul 08 16:35:38.928773 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:38.928328 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" event={"ID":"d5ea3d53-7068-4df8-87be-71e0e876a17d","Type":"ContainerStarted","Data":"93bb5abe21085f54688f363246381f83b91932bb60096cb86b64fc92fab32bae"} Jul 08 16:35:38.954327 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:38.951621 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" podStartSLOduration=1.951601385 podStartE2EDuration="1.951601385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:35:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:35:38.948347207 +0000 UTC m=+1941.495192326" watchObservedRunningTime="2026-07-08 16:35:38.951601385 +0000 UTC m=+1941.498446500" Jul 08 16:35:40.515002 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:40.514969 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:35:47.915660 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:47.915622 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:47.916262 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:47.915889 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:47.928579 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:47.928552 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:47.983200 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:47.983169 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:35:57.517325 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:35:57.517287 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:36:21.324601 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:21.324564 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:36:23.039263 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:23.039232 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2"] Jul 08 16:36:35.312749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:35.312707 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:36:38.060225 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.060179 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" podUID="d5ea3d53-7068-4df8-87be-71e0e876a17d" containerName="main" containerID="cri-o://f5cc3261f4e2d9ffc4c8198bfedab02bf2272764b6bd1ba96f9c1e7e6339a823" gracePeriod=45 Jul 08 16:36:38.215654 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.215619 2563 generic.go:356] "Generic (PLEG): container finished" podID="d5ea3d53-7068-4df8-87be-71e0e876a17d" containerID="f5cc3261f4e2d9ffc4c8198bfedab02bf2272764b6bd1ba96f9c1e7e6339a823" exitCode=0 Jul 08 16:36:38.215835 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.215694 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" event={"ID":"d5ea3d53-7068-4df8-87be-71e0e876a17d","Type":"ContainerDied","Data":"f5cc3261f4e2d9ffc4c8198bfedab02bf2272764b6bd1ba96f9c1e7e6339a823"} Jul 08 16:36:38.319680 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.319617 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:36:38.437129 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437094 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-model-cache\") pod \"d5ea3d53-7068-4df8-87be-71e0e876a17d\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " Jul 08 16:36:38.437325 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437152 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-tmp-dir\") pod \"d5ea3d53-7068-4df8-87be-71e0e876a17d\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " Jul 08 16:36:38.437325 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437174 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-dshm\") pod \"d5ea3d53-7068-4df8-87be-71e0e876a17d\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " Jul 08 16:36:38.437325 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437199 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgzm7\" (UniqueName: \"kubernetes.io/projected/d5ea3d53-7068-4df8-87be-71e0e876a17d-kube-api-access-rgzm7\") pod \"d5ea3d53-7068-4df8-87be-71e0e876a17d\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " Jul 08 16:36:38.437325 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437240 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ea3d53-7068-4df8-87be-71e0e876a17d-tls-certs\") pod \"d5ea3d53-7068-4df8-87be-71e0e876a17d\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " Jul 08 16:36:38.437325 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437318 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-home\") pod \"d5ea3d53-7068-4df8-87be-71e0e876a17d\" (UID: \"d5ea3d53-7068-4df8-87be-71e0e876a17d\") " Jul 08 16:36:38.437589 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437397 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-model-cache" (OuterVolumeSpecName: "model-cache") pod "d5ea3d53-7068-4df8-87be-71e0e876a17d" (UID: "d5ea3d53-7068-4df8-87be-71e0e876a17d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:36:38.437589 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437468 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d5ea3d53-7068-4df8-87be-71e0e876a17d" (UID: "d5ea3d53-7068-4df8-87be-71e0e876a17d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:36:38.437703 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437611 2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-model-cache\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:36:38.437703 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437637 2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-tmp-dir\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:36:38.437703 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.437610 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-home" (OuterVolumeSpecName: "home") pod "d5ea3d53-7068-4df8-87be-71e0e876a17d" (UID: "d5ea3d53-7068-4df8-87be-71e0e876a17d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:36:38.439603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.439571 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-dshm" (OuterVolumeSpecName: "dshm") pod "d5ea3d53-7068-4df8-87be-71e0e876a17d" (UID: "d5ea3d53-7068-4df8-87be-71e0e876a17d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:36:38.439603 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.439583 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ea3d53-7068-4df8-87be-71e0e876a17d-kube-api-access-rgzm7" (OuterVolumeSpecName: "kube-api-access-rgzm7") pod "d5ea3d53-7068-4df8-87be-71e0e876a17d" (UID: "d5ea3d53-7068-4df8-87be-71e0e876a17d"). InnerVolumeSpecName "kube-api-access-rgzm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:36:38.439738 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.439608 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ea3d53-7068-4df8-87be-71e0e876a17d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d5ea3d53-7068-4df8-87be-71e0e876a17d" (UID: "d5ea3d53-7068-4df8-87be-71e0e876a17d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 08 16:36:38.538554 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.538526 2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-dshm\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:36:38.538554 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.538551 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgzm7\" (UniqueName: \"kubernetes.io/projected/d5ea3d53-7068-4df8-87be-71e0e876a17d-kube-api-access-rgzm7\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:36:38.538554 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.538561 2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5ea3d53-7068-4df8-87be-71e0e876a17d-tls-certs\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:36:38.538803 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:38.538571 2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5ea3d53-7068-4df8-87be-71e0e876a17d-home\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:36:39.221721 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:39.221689 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" Jul 08 16:36:39.222155 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:39.221723 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2" event={"ID":"d5ea3d53-7068-4df8-87be-71e0e876a17d","Type":"ContainerDied","Data":"93bb5abe21085f54688f363246381f83b91932bb60096cb86b64fc92fab32bae"} Jul 08 16:36:39.222155 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:39.221767 2563 scope.go:117] "RemoveContainer" containerID="f5cc3261f4e2d9ffc4c8198bfedab02bf2272764b6bd1ba96f9c1e7e6339a823" Jul 08 16:36:39.242391 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:39.242365 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2"] Jul 08 16:36:39.247672 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:39.247651 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-dcd525/test-model-dcd525-kserve-699bffff99-v8xs2"] Jul 08 16:36:40.002170 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:36:40.002132 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5ea3d53-7068-4df8-87be-71e0e876a17d" path="/var/lib/kubelet/pods/d5ea3d53-7068-4df8-87be-71e0e876a17d/volumes" Jul 08 16:37:10.416258 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:10.416220 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:37:13.419228 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:13.419195 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv"] Jul 08 16:37:13.419688 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:13.419451 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" containerID="cri-o://671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186" gracePeriod=30 Jul 08 16:37:14.726929 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:14.726889 2563 patch_prober.go:28] interesting pod/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Jul 08 16:37:14.727324 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:14.726965 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Jul 08 16:37:18.673611 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.673584 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:37:18.790322 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790242 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istiod-ca-cert\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790322 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790278 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tskdt\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-kube-api-access-tskdt\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790322 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790311 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-token\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790336 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-data\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790365 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-podinfo\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790387 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-certs\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790482 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-credential-socket\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790542 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-socket\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790573 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-envoy\") pod \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\" (UID: \"7213606d-e3d2-4e28-92c4-fdd1508f3c15\") " Jul 08 16:37:18.790888 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790756 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:18.790888 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790693 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 16:37:18.790888 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790842 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:18.791038 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790900 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-credential-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:18.791038 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790920 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:18.791038 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.790933 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istiod-ca-cert\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:18.791216 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.791140 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:18.791377 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.791354 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-data" (OuterVolumeSpecName: "istio-data") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:18.793051 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.792977 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-kube-api-access-tskdt" (OuterVolumeSpecName: "kube-api-access-tskdt") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "kube-api-access-tskdt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:37:18.793193 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.793050 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:18.793193 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.793123 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-token" (OuterVolumeSpecName: "istio-token") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:37:18.793193 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.793152 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7213606d-e3d2-4e28-92c4-fdd1508f3c15" (UID: "7213606d-e3d2-4e28-92c4-fdd1508f3c15"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 16:37:18.892086 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.892028 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tskdt\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-kube-api-access-tskdt\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:18.892086 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.892059 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-token\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:18.892086 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.892090 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-data\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:18.892612 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.892097 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-podinfo\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:18.892612 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.892106 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-workload-certs\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:18.892612 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:18.892114 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7213606d-e3d2-4e28-92c4-fdd1508f3c15-istio-envoy\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:19.414982 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.414944 2563 generic.go:356] "Generic (PLEG): container finished" podID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerID="671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186" exitCode=0 Jul 08 16:37:19.415183 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.414995 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" Jul 08 16:37:19.415183 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.415008 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" event={"ID":"7213606d-e3d2-4e28-92c4-fdd1508f3c15","Type":"ContainerDied","Data":"671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186"} Jul 08 16:37:19.415183 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.415050 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv" event={"ID":"7213606d-e3d2-4e28-92c4-fdd1508f3c15","Type":"ContainerDied","Data":"d5851eca34051e1adbf3af0aa1b43f806d03db71dcbf478a988fef89f0dc70be"} Jul 08 16:37:19.415183 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.415092 2563 scope.go:117] "RemoveContainer" containerID="671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186" Jul 08 16:37:19.424491 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.424474 2563 scope.go:117] "RemoveContainer" containerID="62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410" Jul 08 16:37:19.432342 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.432326 2563 scope.go:117] "RemoveContainer" containerID="671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186" Jul 08 16:37:19.432586 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:37:19.432570 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186\": container with ID starting with 671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186 not found: ID does not exist" containerID="671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186" Jul 08 16:37:19.432628 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.432593 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186"} err="failed to get container status \"671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186\": rpc error: code = NotFound desc = could not find container \"671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186\": container with ID starting with 671837092e32872ab6e490503c98276aa9637322bc9c7be54ef99c11261a1186 not found: ID does not exist" Jul 08 16:37:19.432628 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.432609 2563 scope.go:117] "RemoveContainer" containerID="62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410" Jul 08 16:37:19.432818 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:37:19.432802 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410\": container with ID starting with 62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410 not found: ID does not exist" containerID="62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410" Jul 08 16:37:19.432854 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.432823 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410"} err="failed to get container status \"62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410\": rpc error: code = NotFound desc = could not find container \"62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410\": container with ID starting with 62eee0b94eb14687f3c9b317affa1b31a3bafd0869edbbbafbb06abf3e898410 not found: ID does not exist" Jul 08 16:37:19.437686 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.437664 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv"] Jul 08 16:37:19.441327 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:19.441308 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-dcd525-openshift-default-866597774f-rwbvv"] Jul 08 16:37:20.002390 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:20.002354 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" path="/var/lib/kubelet/pods/7213606d-e3d2-4e28-92c4-fdd1508f3c15/volumes" Jul 08 16:37:22.122310 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122275 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z"] Jul 08 16:37:22.122880 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122790 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" Jul 08 16:37:22.122880 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122810 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" Jul 08 16:37:22.122880 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122825 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" Jul 08 16:37:22.122880 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122834 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" Jul 08 16:37:22.122880 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122873 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5ea3d53-7068-4df8-87be-71e0e876a17d" containerName="main" Jul 08 16:37:22.122880 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122882 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5ea3d53-7068-4df8-87be-71e0e876a17d" containerName="main" Jul 08 16:37:22.123213 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122960 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5ea3d53-7068-4df8-87be-71e0e876a17d" containerName="main" Jul 08 16:37:22.123213 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122975 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" Jul 08 16:37:22.123213 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.122985 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7213606d-e3d2-4e28-92c4-fdd1508f3c15" containerName="istio-proxy" Jul 08 16:37:22.125911 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.125890 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.129737 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.129712 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 16:37:22.129898 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.129792 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-63af7a-openshift-default-dockercfg-chdsg\"" Jul 08 16:37:22.138233 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.138203 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z"] Jul 08 16:37:22.220556 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220519 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-credential-socket\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.220733 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220562 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjs5d\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-kube-api-access-gjs5d\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.220733 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220646 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-data\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.220733 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220662 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5506754e-2934-4c7d-8398-92134be1c913-istio-podinfo\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.220733 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220684 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-certs\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.220733 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220708 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-istio-token\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.220942 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220785 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-envoy\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.220942 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220828 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-socket\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.220942 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.220859 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5506754e-2934-4c7d-8398-92134be1c913-istiod-ca-cert\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.321984 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.321947 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjs5d\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-kube-api-access-gjs5d\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322218 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322003 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-data\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322218 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322022 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5506754e-2934-4c7d-8398-92134be1c913-istio-podinfo\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322218 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322042 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-certs\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322218 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322210 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-istio-token\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322483 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322258 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-envoy\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322483 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322314 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-socket\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322483 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322340 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5506754e-2934-4c7d-8398-92134be1c913-istiod-ca-cert\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322483 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322407 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-credential-socket\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322483 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322410 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-certs\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322483 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322439 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-data\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322781 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322677 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-socket\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.322781 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.322700 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-credential-socket\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.323183 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.323159 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5506754e-2934-4c7d-8398-92134be1c913-istiod-ca-cert\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.324835 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.324805 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-envoy\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.325031 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.325017 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5506754e-2934-4c7d-8398-92134be1c913-istio-podinfo\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.333256 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.333237 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-istio-token\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.335567 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.335542 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjs5d\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-kube-api-access-gjs5d\") pod \"e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.436749 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.436717 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:22.577764 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.577734 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z"] Jul 08 16:37:22.579609 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:37:22.579582 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5506754e_2934_4c7d_8398_92134be1c913.slice/crio-c1cc35eaee2acc35b9bffef63b1b24fca84e909023e2a2aa994c3e53c97982a1 WatchSource:0}: Error finding container c1cc35eaee2acc35b9bffef63b1b24fca84e909023e2a2aa994c3e53c97982a1: Status 404 returned error can't find the container with id c1cc35eaee2acc35b9bffef63b1b24fca84e909023e2a2aa994c3e53c97982a1 Jul 08 16:37:22.581863 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.581827 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:37:22.581923 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.581896 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:37:22.581958 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:22.581926 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:37:23.437282 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:23.437250 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" event={"ID":"5506754e-2934-4c7d-8398-92134be1c913","Type":"ContainerStarted","Data":"418c9d5a8c533c523d24b78b9821f728fa5150e1dfdab41cf20d7e401591962e"} Jul 08 16:37:23.437979 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:23.437309 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" event={"ID":"5506754e-2934-4c7d-8398-92134be1c913","Type":"ContainerStarted","Data":"c1cc35eaee2acc35b9bffef63b1b24fca84e909023e2a2aa994c3e53c97982a1"} Jul 08 16:37:23.461107 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:23.461026 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" podStartSLOduration=1.4610025819999999 podStartE2EDuration="1.461002582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:37:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:37:23.459054891 +0000 UTC m=+2046.005900008" watchObservedRunningTime="2026-07-08 16:37:23.461002582 +0000 UTC m=+2046.007847698" Jul 08 16:37:24.437128 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:24.437061 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:24.442300 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:24.442275 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:25.030686 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:25.030650 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:37:25.446149 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:25.446113 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:25.447260 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:25.447243 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:41.214446 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:41.214417 2563 patch_prober.go:28] interesting pod/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": read tcp 10.132.0.2:55112->10.132.0.42:15021: read: connection reset by peer" start-of-body= Jul 08 16:37:41.214809 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:41.214523 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": read tcp 10.132.0.2:55112->10.132.0.42:15021: read: connection reset by peer" Jul 08 16:37:41.522990 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:41.522897 2563 generic.go:356] "Generic (PLEG): container finished" podID="5506754e-2934-4c7d-8398-92134be1c913" containerID="418c9d5a8c533c523d24b78b9821f728fa5150e1dfdab41cf20d7e401591962e" exitCode=0 Jul 08 16:37:41.522990 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:41.522951 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" event={"ID":"5506754e-2934-4c7d-8398-92134be1c913","Type":"ContainerDied","Data":"418c9d5a8c533c523d24b78b9821f728fa5150e1dfdab41cf20d7e401591962e"} Jul 08 16:37:41.523401 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:41.523387 2563 scope.go:117] "RemoveContainer" containerID="418c9d5a8c533c523d24b78b9821f728fa5150e1dfdab41cf20d7e401591962e" Jul 08 16:37:41.524440 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:41.524412 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:37:41.524503 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:41.524487 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:37:41.524545 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:41.524522 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:37:42.216697 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:42.216662 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:37:42.529246 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:42.529160 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" event={"ID":"5506754e-2934-4c7d-8398-92134be1c913","Type":"ContainerStarted","Data":"e84a105335d4c61340621a5123dde6051aee5b59653e6ad9aaeca1ab8d83cfc5"} Jul 08 16:37:43.437792 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:43.437747 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:43.438274 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:43.437967 2563 patch_prober.go:28] interesting pod/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 08 16:37:43.438274 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:43.438015 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 08 16:37:44.437424 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:44.437381 2563 patch_prober.go:28] interesting pod/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 08 16:37:44.437615 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:44.437439 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 08 16:37:45.438044 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:45.438002 2563 patch_prober.go:28] interesting pod/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" start-of-body= Jul 08 16:37:45.438499 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:45.438100 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.42:15021/healthz/ready\": dial tcp 10.132.0.42:15021: connect: connection refused" Jul 08 16:37:46.490863 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:46.490817 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:46.491358 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:46.491335 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:46.492998 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:46.492976 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:52.440697 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:52.440660 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z"] Jul 08 16:37:52.441196 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:52.440880 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" containerID="cri-o://e84a105335d4c61340621a5123dde6051aee5b59653e6ad9aaeca1ab8d83cfc5" gracePeriod=30 Jul 08 16:37:56.429348 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:56.429308 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:37:57.604708 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.604679 2563 generic.go:356] "Generic (PLEG): container finished" podID="5506754e-2934-4c7d-8398-92134be1c913" containerID="e84a105335d4c61340621a5123dde6051aee5b59653e6ad9aaeca1ab8d83cfc5" exitCode=0 Jul 08 16:37:57.605107 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.604740 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" event={"ID":"5506754e-2934-4c7d-8398-92134be1c913","Type":"ContainerDied","Data":"e84a105335d4c61340621a5123dde6051aee5b59653e6ad9aaeca1ab8d83cfc5"} Jul 08 16:37:57.605107 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.604792 2563 scope.go:117] "RemoveContainer" containerID="418c9d5a8c533c523d24b78b9821f728fa5150e1dfdab41cf20d7e401591962e" Jul 08 16:37:57.703344 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.703280 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:57.748632 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.748594 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gjs5d\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-kube-api-access-gjs5d\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.748827 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.748647 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5506754e-2934-4c7d-8398-92134be1c913-istio-podinfo\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.748827 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.748767 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-envoy\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.748827 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.748803 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-data\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.748994 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.748855 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-credential-socket\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.748994 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.748924 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-certs\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.748994 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.748949 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5506754e-2934-4c7d-8398-92134be1c913-istiod-ca-cert\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.748994 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.748986 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-istio-token\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.749234 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.749008 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-socket\") pod \"5506754e-2934-4c7d-8398-92134be1c913\" (UID: \"5506754e-2934-4c7d-8398-92134be1c913\") " Jul 08 16:37:57.749304 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.749277 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:57.749608 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.749350 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5506754e-2934-4c7d-8398-92134be1c913-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 16:37:57.749608 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.749408 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-certs\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:57.749608 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.749502 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:57.749608 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.749524 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:57.750257 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.750232 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-data" (OuterVolumeSpecName: "istio-data") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:57.751121 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.751091 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:37:57.751210 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.751130 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-istio-token" (OuterVolumeSpecName: "istio-token") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:37:57.751319 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.751305 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-kube-api-access-gjs5d" (OuterVolumeSpecName: "kube-api-access-gjs5d") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "kube-api-access-gjs5d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:37:57.751453 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.751439 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5506754e-2934-4c7d-8398-92134be1c913-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5506754e-2934-4c7d-8398-92134be1c913" (UID: "5506754e-2934-4c7d-8398-92134be1c913"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 16:37:57.850824 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.850789 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-data\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:57.850824 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.850818 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-credential-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:57.850824 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.850829 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5506754e-2934-4c7d-8398-92134be1c913-istiod-ca-cert\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:57.851101 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.850840 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-istio-token\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:57.851101 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.850850 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-workload-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:57.851101 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.850857 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjs5d\" (UniqueName: \"kubernetes.io/projected/5506754e-2934-4c7d-8398-92134be1c913-kube-api-access-gjs5d\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:57.851101 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.850866 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5506754e-2934-4c7d-8398-92134be1c913-istio-podinfo\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:57.851101 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:57.850874 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5506754e-2934-4c7d-8398-92134be1c913-istio-envoy\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:37:58.617108 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:58.617052 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" event={"ID":"5506754e-2934-4c7d-8398-92134be1c913","Type":"ContainerDied","Data":"c1cc35eaee2acc35b9bffef63b1b24fca84e909023e2a2aa994c3e53c97982a1"} Jul 08 16:37:58.617572 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:58.617087 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z" Jul 08 16:37:58.617572 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:58.617147 2563 scope.go:117] "RemoveContainer" containerID="e84a105335d4c61340621a5123dde6051aee5b59653e6ad9aaeca1ab8d83cfc5" Jul 08 16:37:58.636024 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:58.635999 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z"] Jul 08 16:37:58.640799 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:37:58.640777 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-63af7a-openshift-default-65f6998db9-l2k9z"] Jul 08 16:38:00.002550 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:00.002506 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5506754e-2934-4c7d-8398-92134be1c913" path="/var/lib/kubelet/pods/5506754e-2934-4c7d-8398-92134be1c913/volumes" Jul 08 16:38:01.158500 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.158472 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t"] Jul 08 16:38:01.158954 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.158846 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" Jul 08 16:38:01.158954 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.158859 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" Jul 08 16:38:01.158954 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.158878 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" Jul 08 16:38:01.158954 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.158886 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" Jul 08 16:38:01.158954 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.158944 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" Jul 08 16:38:01.162149 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.162127 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.167696 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.167674 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Jul 08 16:38:01.167871 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.167850 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-5494d3-openshift-default-dockercfg-n9765\"" Jul 08 16:38:01.185313 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.185289 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t"] Jul 08 16:38:01.282212 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282176 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-socket\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.282424 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282221 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-credential-socket\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.282424 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282254 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-istio-token\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.282424 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282302 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-certs\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.282424 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282328 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5971cc-c972-415f-939e-ea5113c42d95-istiod-ca-cert\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.282424 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282347 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9zt6\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-kube-api-access-j9zt6\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.282424 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282393 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-envoy\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.282424 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282414 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-data\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.282788 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.282433 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5971cc-c972-415f-939e-ea5113c42d95-istio-podinfo\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.383573 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383533 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-socket\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.383774 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383583 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-credential-socket\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.383774 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383616 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-istio-token\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.383774 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383667 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-certs\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.383774 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383693 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5971cc-c972-415f-939e-ea5113c42d95-istiod-ca-cert\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383831 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9zt6\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-kube-api-access-j9zt6\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383938 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-envoy\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383948 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-socket\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383965 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-data\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384007 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.383994 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5971cc-c972-415f-939e-ea5113c42d95-istio-podinfo\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384291 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.384053 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-credential-socket\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384351 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.384288 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-certs\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384454 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.384428 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-data\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.384517 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.384467 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5971cc-c972-415f-939e-ea5113c42d95-istiod-ca-cert\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.386604 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.386578 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5971cc-c972-415f-939e-ea5113c42d95-istio-podinfo\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.386777 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.386759 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-envoy\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.392905 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.392882 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-istio-token\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.393169 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.393058 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9zt6\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-kube-api-access-j9zt6\") pod \"e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.473253 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.473156 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:01.623436 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.623412 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t"] Jul 08 16:38:01.625489 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:38:01.625457 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5971cc_c972_415f_939e_ea5113c42d95.slice/crio-04a3ab3748996f6e78c377c7f3b23896e86e4fb560f4218f6f06bb2b6be7b694 WatchSource:0}: Error finding container 04a3ab3748996f6e78c377c7f3b23896e86e4fb560f4218f6f06bb2b6be7b694: Status 404 returned error can't find the container with id 04a3ab3748996f6e78c377c7f3b23896e86e4fb560f4218f6f06bb2b6be7b694 Jul 08 16:38:01.627810 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.627769 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:38:01.627896 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.627859 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:38:01.627934 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.627904 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:38:01.633744 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:01.633716 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" event={"ID":"ba5971cc-c972-415f-939e-ea5113c42d95","Type":"ContainerStarted","Data":"04a3ab3748996f6e78c377c7f3b23896e86e4fb560f4218f6f06bb2b6be7b694"} Jul 08 16:38:02.640423 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:02.640388 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" event={"ID":"ba5971cc-c972-415f-939e-ea5113c42d95","Type":"ContainerStarted","Data":"ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822"} Jul 08 16:38:02.664475 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:02.664397 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" podStartSLOduration=1.664373388 podStartE2EDuration="1.664373388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-08 16:38:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:38:02.662681757 +0000 UTC m=+2085.209526871" watchObservedRunningTime="2026-07-08 16:38:02.664373388 +0000 UTC m=+2085.211218502" Jul 08 16:38:03.474050 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:03.474017 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:03.478943 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:03.478917 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:03.645300 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:03.645268 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:03.646315 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:03.646291 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:10.614140 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:10.614096 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:38:18.077910 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.077884 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:38:18.079695 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.079674 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:38:18.647024 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.646944 2563 patch_prober.go:28] interesting pod/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 08 16:38:18.647024 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.647005 2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 08 16:38:18.720231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.720148 2563 generic.go:356] "Generic (PLEG): container finished" podID="ba5971cc-c972-415f-939e-ea5113c42d95" containerID="ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822" exitCode=0 Jul 08 16:38:18.720231 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.720188 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" event={"ID":"ba5971cc-c972-415f-939e-ea5113c42d95","Type":"ContainerDied","Data":"ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822"} Jul 08 16:38:18.720632 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.720614 2563 scope.go:117] "RemoveContainer" containerID="ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822" Jul 08 16:38:18.721617 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.721588 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:38:18.721681 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.721657 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:38:18.721718 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:18.721696 2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"} Jul 08 16:38:19.726810 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:19.726777 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" event={"ID":"ba5971cc-c972-415f-939e-ea5113c42d95","Type":"ContainerStarted","Data":"db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19"} Jul 08 16:38:20.473832 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:20.473787 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:20.474044 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:20.473987 2563 patch_prober.go:28] interesting pod/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 08 16:38:20.474140 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:20.474043 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 08 16:38:21.474020 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:21.473977 2563 patch_prober.go:28] interesting pod/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 08 16:38:21.474718 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:21.474684 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 08 16:38:22.474342 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:22.474302 2563 patch_prober.go:28] interesting pod/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 08 16:38:22.474950 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:22.474917 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 08 16:38:23.474435 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:23.474388 2563 patch_prober.go:28] interesting pod/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 08 16:38:23.474941 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:23.474457 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 08 16:38:24.474307 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:24.474270 2563 patch_prober.go:28] interesting pod/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body= Jul 08 16:38:24.474614 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:24.474584 2563 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" Jul 08 16:38:25.478211 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:25.478177 2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:25.478594 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:25.478437 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:38:25.479210 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:38:25.479192 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:41:17.912264 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:17.912222 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:41:27.871752 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:27.871717 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t"] Jul 08 16:41:27.872319 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:27.872288 2563 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" containerID="cri-o://db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19" gracePeriod=30 Jul 08 16:41:30.512205 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:30.512166 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-7xjhr"] Jul 08 16:41:33.129604 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.129578 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:41:33.256593 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256558 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-credential-socket\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.256593 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256596 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-istio-token\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.256842 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256622 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j9zt6\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-kube-api-access-j9zt6\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.256842 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256647 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-data\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.256842 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256705 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5971cc-c972-415f-939e-ea5113c42d95-istio-podinfo\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.256842 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256788 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-envoy\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.256842 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256824 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5971cc-c972-415f-939e-ea5113c42d95-istiod-ca-cert\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.257133 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256856 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-certs\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.257133 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.256903 2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-socket\") pod \"ba5971cc-c972-415f-939e-ea5113c42d95\" (UID: \"ba5971cc-c972-415f-939e-ea5113c42d95\") " Jul 08 16:41:33.257478 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.257240 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:41:33.257478 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.257249 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:41:33.257478 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.257344 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:41:33.257661 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.257482 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba5971cc-c972-415f-939e-ea5113c42d95-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 08 16:41:33.257764 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.257721 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-data" (OuterVolumeSpecName: "istio-data") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:41:33.259378 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.259353 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ba5971cc-c972-415f-939e-ea5113c42d95-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Jul 08 16:41:33.259624 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.259583 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 08 16:41:33.259624 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.259592 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-istio-token" (OuterVolumeSpecName: "istio-token") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:41:33.259771 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.259656 2563 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-kube-api-access-j9zt6" (OuterVolumeSpecName: "kube-api-access-j9zt6") pod "ba5971cc-c972-415f-939e-ea5113c42d95" (UID: "ba5971cc-c972-415f-939e-ea5113c42d95"). InnerVolumeSpecName "kube-api-access-j9zt6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 08 16:41:33.357639 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357601 2563 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-certs\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.357639 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357632 2563 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-workload-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.357639 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357640 2563 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-credential-socket\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.357639 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357650 2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-istio-token\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.357919 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357658 2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9zt6\" (UniqueName: \"kubernetes.io/projected/ba5971cc-c972-415f-939e-ea5113c42d95-kube-api-access-j9zt6\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.357919 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357669 2563 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-data\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.357919 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357677 2563 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ba5971cc-c972-415f-939e-ea5113c42d95-istio-podinfo\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.357919 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357684 2563 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ba5971cc-c972-415f-939e-ea5113c42d95-istio-envoy\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.357919 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.357692 2563 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ba5971cc-c972-415f-939e-ea5113c42d95-istiod-ca-cert\") on node \"ip-10-0-141-121.ec2.internal\" DevicePath \"\"" Jul 08 16:41:33.626445 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.626357 2563 generic.go:356] "Generic (PLEG): container finished" podID="ba5971cc-c972-415f-939e-ea5113c42d95" containerID="db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19" exitCode=0 Jul 08 16:41:33.626445 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.626409 2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" Jul 08 16:41:33.626445 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.626416 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" event={"ID":"ba5971cc-c972-415f-939e-ea5113c42d95","Type":"ContainerDied","Data":"db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19"} Jul 08 16:41:33.626676 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.626469 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t" event={"ID":"ba5971cc-c972-415f-939e-ea5113c42d95","Type":"ContainerDied","Data":"04a3ab3748996f6e78c377c7f3b23896e86e4fb560f4218f6f06bb2b6be7b694"} Jul 08 16:41:33.626676 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.626499 2563 scope.go:117] "RemoveContainer" containerID="db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19" Jul 08 16:41:33.635804 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.635788 2563 scope.go:117] "RemoveContainer" containerID="ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822" Jul 08 16:41:33.643412 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.643395 2563 scope.go:117] "RemoveContainer" containerID="db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19" Jul 08 16:41:33.643652 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:41:33.643632 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19\": container with ID starting with db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19 not found: ID does not exist" containerID="db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19" Jul 08 16:41:33.643703 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.643660 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19"} err="failed to get container status \"db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19\": rpc error: code = NotFound desc = could not find container \"db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19\": container with ID starting with db044885254e1941c81c931a0b5cc2cd9394ce31363c2caf4982f0fc519b3f19 not found: ID does not exist" Jul 08 16:41:33.643703 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.643676 2563 scope.go:117] "RemoveContainer" containerID="ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822" Jul 08 16:41:33.643890 ip-10-0-141-121 kubenswrapper[2563]: E0708 16:41:33.643875 2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822\": container with ID starting with ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822 not found: ID does not exist" containerID="ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822" Jul 08 16:41:33.643934 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.643892 2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822"} err="failed to get container status \"ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822\": rpc error: code = NotFound desc = could not find container \"ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822\": container with ID starting with ea85dd6ba0266d4b2fe4769cbe18a36ba2efa996e863856d28db8de65cbf3822 not found: ID does not exist" Jul 08 16:41:33.651752 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.651728 2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t"] Jul 08 16:41:33.658058 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:33.658028 2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-5494d3-openshift-default-5c45ddbd8c-jzw6t"] Jul 08 16:41:34.002764 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:34.002726 2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" path="/var/lib/kubelet/pods/ba5971cc-c972-415f-939e-ea5113c42d95/volumes" Jul 08 16:41:34.415649 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:34.415620 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bf8d94745-bqdc5_8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4/copy-manifests/0.log" Jul 08 16:41:34.429519 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:34.429494 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bf8d94745-bqdc5_8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4/manager/0.log" Jul 08 16:41:36.104458 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:36.104426 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bf74w_d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5/manager/0.log" Jul 08 16:41:36.464282 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:36.464246 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jkw5p_86c469d2-ab9f-4e5c-ab8a-9666512275a8/manager/0.log" Jul 08 16:41:36.578236 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:36.578201 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7xjhr_6e28216f-f511-43c6-a7cf-3c7d2ba77f6e/limitador/0.log" Jul 08 16:41:37.656897 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:37.656867 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-5ll5d_b241870a-38a7-4fa6-8549-82be003e21f4/payload-pre-processing/0.log" Jul 08 16:41:37.902731 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:37.902704 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f96688f5-4vp7w_e1292322-1790-4f8c-83ed-a18cae7ef53c/router/0.log" Jul 08 16:41:38.256674 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:38.256644 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-rzsch_651ef463-7509-4d65-806f-7cc87bccaaa6/main/0.log" Jul 08 16:41:38.616157 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:38.616056 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86db69sfv_4f754de9-d72e-4387-b33b-34168c7cd116/main/0.log" Jul 08 16:41:38.852025 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:38.851997 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-tk4qw_89de82cb-7bf0-4ab2-94f4-a636eac46462/main/0.log" Jul 08 16:41:46.241194 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:46.241164 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-wvsd9_d003ff65-18de-4f6a-91b8-5dd4c2db3e12/global-pull-secret-syncer/0.log" Jul 08 16:41:46.363577 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:46.363543 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zkbjr_2c18d608-ccd9-4075-9224-2b9a50cea4f7/konnectivity-agent/0.log" Jul 08 16:41:46.435962 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:46.435930 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-121.ec2.internal_0f73b543c0aedc66b9e736106d082533/haproxy/0.log" Jul 08 16:41:51.968211 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:51.968179 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-bf74w_d9bbaa9a-aa6b-4a9b-82e3-68feff48f2b5/manager/0.log" Jul 08 16:41:52.176950 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:52.176911 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-jkw5p_86c469d2-ab9f-4e5c-ab8a-9666512275a8/manager/0.log" Jul 08 16:41:52.219160 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:52.219056 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-7xjhr_6e28216f-f511-43c6-a7cf-3c7d2ba77f6e/limitador/0.log" Jul 08 16:41:54.111740 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:54.111705 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-8x9d5_b6ea2261-c7e6-49b0-84fb-0a7a3c736f08/cluster-monitoring-operator/0.log" Jul 08 16:41:54.366452 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:54.366379 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-stpsb_0f6570cf-85d1-47bc-a87e-d091b30452e0/node-exporter/0.log" Jul 08 16:41:54.389009 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:54.388980 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-stpsb_0f6570cf-85d1-47bc-a87e-d091b30452e0/kube-rbac-proxy/0.log" Jul 08 16:41:54.411675 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:54.411655 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-stpsb_0f6570cf-85d1-47bc-a87e-d091b30452e0/init-textfile/0.log" Jul 08 16:41:56.229606 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.229582 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-jhwzx_bfdb2476-acb0-483c-8612-85e59e2d5303/networking-console-plugin/0.log" Jul 08 16:41:56.608593 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.608511 2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k"] Jul 08 16:41:56.608865 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.608853 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" Jul 08 16:41:56.608906 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.608867 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" Jul 08 16:41:56.608906 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.608883 2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" Jul 08 16:41:56.608906 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.608889 2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" Jul 08 16:41:56.609005 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.608952 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5506754e-2934-4c7d-8398-92134be1c913" containerName="istio-proxy" Jul 08 16:41:56.609005 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.608964 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" Jul 08 16:41:56.609005 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.608971 2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ba5971cc-c972-415f-939e-ea5113c42d95" containerName="istio-proxy" Jul 08 16:41:56.612136 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.612114 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.614598 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.614579 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qvbgl\"/\"kube-root-ca.crt\"" Jul 08 16:41:56.615502 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.615488 2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-qvbgl\"/\"default-dockercfg-2sphb\"" Jul 08 16:41:56.615561 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.615535 2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-qvbgl\"/\"openshift-service-ca.crt\"" Jul 08 16:41:56.622258 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.622234 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k"] Jul 08 16:41:56.756352 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.756322 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-podres\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.756352 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.756358 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-sys\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.756575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.756422 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-proc\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.756575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.756457 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-lib-modules\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.756575 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.756528 2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2vl6\" (UniqueName: \"kubernetes.io/projected/192e2a61-b6e5-4a02-ad51-022e23fce1f4-kube-api-access-c2vl6\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857168 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857134 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-proc\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857168 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857170 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-lib-modules\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857385 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857228 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2vl6\" (UniqueName: \"kubernetes.io/projected/192e2a61-b6e5-4a02-ad51-022e23fce1f4-kube-api-access-c2vl6\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857385 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857288 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-podres\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857385 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857310 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-lib-modules\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857385 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857340 2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-sys\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857385 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857353 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-proc\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857546 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857393 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-podres\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.857546 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.857406 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/192e2a61-b6e5-4a02-ad51-022e23fce1f4-sys\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.867598 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.867536 2563 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2vl6\" (UniqueName: \"kubernetes.io/projected/192e2a61-b6e5-4a02-ad51-022e23fce1f4-kube-api-access-c2vl6\") pod \"perf-node-gather-daemonset-fnj9k\" (UID: \"192e2a61-b6e5-4a02-ad51-022e23fce1f4\") " pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:56.912281 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.912256 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/1.log" Jul 08 16:41:56.916453 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.916434 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-wp8nf_1b73f265-f5e7-4aaa-a960-ea10101d0d55/console-operator/2.log" Jul 08 16:41:56.922764 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:56.922743 2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:57.058473 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:57.058384 2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k"] Jul 08 16:41:57.060747 ip-10-0-141-121 kubenswrapper[2563]: W0708 16:41:57.060722 2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod192e2a61_b6e5_4a02_ad51_022e23fce1f4.slice/crio-cc98bcb226c9e66f35f6275ad95db3643df7279962b5b6b63863a9af44b1450a WatchSource:0}: Error finding container cc98bcb226c9e66f35f6275ad95db3643df7279962b5b6b63863a9af44b1450a: Status 404 returned error can't find the container with id cc98bcb226c9e66f35f6275ad95db3643df7279962b5b6b63863a9af44b1450a Jul 08 16:41:57.513997 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:57.513969 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-685bfb5f64-gxwws_28f5cc45-4fcd-4a06-b6dd-00a1165763a3/download-server/0.log" Jul 08 16:41:57.739704 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:57.739670 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" event={"ID":"192e2a61-b6e5-4a02-ad51-022e23fce1f4","Type":"ContainerStarted","Data":"71c170f7810dce4d72696f03d96d57412d93cb625aea274ca5dfe1868e7bb608"} Jul 08 16:41:57.739907 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:57.739750 2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" event={"ID":"192e2a61-b6e5-4a02-ad51-022e23fce1f4","Type":"ContainerStarted","Data":"cc98bcb226c9e66f35f6275ad95db3643df7279962b5b6b63863a9af44b1450a"} Jul 08 16:41:57.739907 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:57.739789 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:41:57.756833 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:57.756785 2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" podStartSLOduration=1.756771064 podStartE2EDuration="1.756771064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-08 16:41:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-08 16:41:57.75473871 +0000 UTC m=+2320.301583824" watchObservedRunningTime="2026-07-08 16:41:57.756771064 +0000 UTC m=+2320.303616178" Jul 08 16:41:58.145694 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:58.145669 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-qslqf_474c28fe-7e30-4d21-a705-1fc25a2ee18f/volume-data-source-validator/0.log" Jul 08 16:41:59.272742 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:59.272713 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhgft_54a18c97-d257-42c6-953a-115296d85fc5/dns/0.log" Jul 08 16:41:59.295005 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:59.294979 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zhgft_54a18c97-d257-42c6-953a-115296d85fc5/kube-rbac-proxy/0.log" Jul 08 16:41:59.371975 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:59.371949 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jnp8q_e6057259-93e1-4083-b7c0-863ea9fff32c/dns-node-resolver/0.log" Jul 08 16:41:59.907195 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:41:59.907168 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-27pnz_849b9b0e-e00b-4793-aba1-38110bc88117/node-ca/0.log" Jul 08 16:42:01.188289 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:01.188264 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-5ll5d_b241870a-38a7-4fa6-8549-82be003e21f4/payload-pre-processing/0.log" Jul 08 16:42:01.296288 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:01.296249 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f96688f5-4vp7w_e1292322-1790-4f8c-83ed-a18cae7ef53c/router/0.log" Jul 08 16:42:01.981848 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:01.981819 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-qthjk_49d96c6c-2207-4d00-ab5e-bf123f948a0f/serve-healthcheck-canary/0.log" Jul 08 16:42:02.518170 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:02.518125 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-vqmfn_76bba082-a1e8-4954-9011-9c6209aa5ae6/insights-operator/0.log" Jul 08 16:42:02.519001 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:02.518972 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-vqmfn_76bba082-a1e8-4954-9011-9c6209aa5ae6/insights-operator/1.log" Jul 08 16:42:02.684451 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:02.684423 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmrh8_65d04972-2959-4b2b-b6b7-5f2ba538cc1d/kube-rbac-proxy/0.log" Jul 08 16:42:02.706206 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:02.706180 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmrh8_65d04972-2959-4b2b-b6b7-5f2ba538cc1d/exporter/0.log" Jul 08 16:42:02.730640 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:02.730610 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vmrh8_65d04972-2959-4b2b-b6b7-5f2ba538cc1d/extractor/0.log" Jul 08 16:42:03.754876 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:03.754853 2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-qvbgl/perf-node-gather-daemonset-fnj9k" Jul 08 16:42:05.582666 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:05.582590 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bf8d94745-bqdc5_8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4/manager/0.log" Jul 08 16:42:05.610642 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:05.610618 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-bf8d94745-bqdc5_8b0fdb5b-7ab1-456e-bc26-2f9bc5a7b3a4/copy-manifests/0.log" Jul 08 16:42:07.037499 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:07.037461 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-7bf555d8d4-45vzp_6c5a02b2-00f4-4437-a1d8-4519a49dca9f/manager/0.log" Jul 08 16:42:07.304170 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:07.304140 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-5ll5d_b241870a-38a7-4fa6-8549-82be003e21f4/payload-pre-processing/0.log" Jul 08 16:42:07.390818 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:07.390788 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f96688f5-4vp7w_e1292322-1790-4f8c-83ed-a18cae7ef53c/router/0.log" Jul 08 16:42:08.853907 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:08.853875 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7964457696-5ll5d_b241870a-38a7-4fa6-8549-82be003e21f4/payload-pre-processing/0.log" Jul 08 16:42:08.946049 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:08.946024 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-75f96688f5-4vp7w_e1292322-1790-4f8c-83ed-a18cae7ef53c/router/0.log" Jul 08 16:42:12.416148 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:12.416051 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-xd8z7_0923cb15-53d6-4f05-a743-efbca787729d/migrator/0.log" Jul 08 16:42:12.452492 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:12.452465 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-d57d58cc9-xd8z7_0923cb15-53d6-4f05-a743-efbca787729d/graceful-termination/0.log" Jul 08 16:42:12.825906 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:12.825820 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-tdhsf_7f660b8b-1a38-4e75-b353-11842c393317/kube-storage-version-migrator-operator/1.log" Jul 08 16:42:12.826703 ip-10-0-141-121 kubenswrapper[2563]: I0708 16:42:12.826685 2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-846bf668f7-tdhsf_7f660b8b-1a38-4e75-b353-11842c393317/kube-storage-version-migrator-operator/0.log"