Jul 03 11:12:16.194031 ip-10-0-141-89 systemd[1]: Starting Kubernetes Kubelet... Jul 03 11:12:16.656225 ip-10-0-141-89 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 11:12:16.656225 ip-10-0-141-89 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Jul 03 11:12:16.656225 ip-10-0-141-89 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 11:12:16.656225 ip-10-0-141-89 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI. Jul 03 11:12:16.656225 ip-10-0-141-89 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Jul 03 11:12:16.657618 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.657183 2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime" Jul 03 11:12:16.665000 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.664972 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 11:12:16.665000 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.664994 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 11:12:16.665000 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665001 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 11:12:16.665000 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665005 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 11:12:16.665000 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665009 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 11:12:16.665000 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665012 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665015 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665020 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665023 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665026 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665029 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665032 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665035 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665046 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665049 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665055 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665057 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665060 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665063 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665065 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665068 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665070 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665073 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665076 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665078 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 11:12:16.665250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665081 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665083 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665086 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665094 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665098 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665101 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665104 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665107 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665109 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665112 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665114 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665117 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665119 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665122 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665126 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665132 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665136 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665140 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665143 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 11:12:16.665746 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665146 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665149 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665157 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665160 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665162 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665166 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665168 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665171 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665176 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665179 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665182 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665184 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665187 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665189 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665199 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665202 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665204 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665208 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665210 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 11:12:16.666229 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665213 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665216 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665221 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665223 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665226 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665228 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665231 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665233 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665236 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665239 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665242 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665244 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665247 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665249 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665254 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665257 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665265 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665269 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665272 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665275 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 11:12:16.666742 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665277 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665280 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665282 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665285 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665288 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665291 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665296 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665299 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665301 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665304 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665308 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665310 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665313 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665315 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665320 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 11:12:16.667212 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.665324 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 11:12:16.668098 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668080 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668100 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668104 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668107 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668110 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668112 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668115 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668118 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668120 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668122 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668125 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668127 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668130 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668133 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668135 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668138 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668140 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 11:12:16.668136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668143 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668146 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668149 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668151 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668154 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668157 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668159 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668161 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668165 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668167 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668170 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668172 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668175 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668177 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668179 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668182 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668184 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668187 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668189 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668192 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 11:12:16.668605 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668195 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668200 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668203 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668208 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668211 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668214 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668216 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668220 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668222 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668225 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668227 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668230 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668232 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668236 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668239 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668242 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668244 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668247 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668249 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 11:12:16.669144 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668251 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668254 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668256 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668258 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668261 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668263 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668266 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668268 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668270 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668273 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668275 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668277 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668283 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668285 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668287 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668290 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668292 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668295 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668297 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668299 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 11:12:16.669615 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668302 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668304 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668306 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668309 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668311 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668313 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668315 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668319 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668321 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668324 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668326 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668329 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668331 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668333 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668336 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668338 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668340 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668343 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668345 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668347 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 11:12:16.670122 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668350 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668353 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668355 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668792 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668799 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668802 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668804 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668807 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668810 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668812 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668815 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668817 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668820 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668823 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668825 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668828 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668830 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668833 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668836 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 11:12:16.670612 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668839 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668842 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668845 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668848 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668851 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668853 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668856 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668859 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668861 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668864 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668866 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668868 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668871 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668873 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668876 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668878 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668880 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668883 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668887 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668891 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 11:12:16.671097 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668893 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668896 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668898 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668901 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668903 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668906 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668908 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668912 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668915 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668918 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668922 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668924 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668928 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668931 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668934 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668937 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668940 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668942 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668945 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668947 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 03 11:12:16.671588 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668949 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668952 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668955 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668958 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668961 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668964 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668966 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668969 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668971 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668974 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668976 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668979 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668981 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668983 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668986 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668988 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668991 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668993 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668995 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 11:12:16.672110 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.668998 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669000 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669002 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669005 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669007 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669010 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669012 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669015 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669020 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669022 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669025 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669027 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669030 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669032 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669035 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669037 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669039 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669042 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669044 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669046 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 11:12:16.672574 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669049 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669051 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669053 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669056 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669137 2581 flags.go:64] FLAG: --address="0.0.0.0" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669145 2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669151 2581 flags.go:64] FLAG: --anonymous-auth="true" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669155 2581 flags.go:64] FLAG: --application-metrics-count-limit="100" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669159 2581 flags.go:64] FLAG: --authentication-token-webhook="false" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669162 2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669167 2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669171 2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669174 2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669177 2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669181 2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669184 2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669188 2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669190 2581 flags.go:64] FLAG: --cgroup-root="" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669193 2581 flags.go:64] FLAG: --cgroups-per-qos="true" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669196 2581 flags.go:64] FLAG: --client-ca-file="" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669200 2581 flags.go:64] FLAG: --cloud-provider="external" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669203 2581 flags.go:64] FLAG: --cluster-dns="[]" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669208 2581 flags.go:64] FLAG: --cluster-domain="" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669211 2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Jul 03 11:12:16.673108 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669216 2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669219 2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669223 2581 flags.go:64] FLAG: --container-log-max-files="5" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669226 2581 flags.go:64] FLAG: --container-log-max-size="10Mi" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669229 2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669232 2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669236 2581 flags.go:64] FLAG: --containerd-namespace="k8s.io" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669239 2581 flags.go:64] FLAG: --contention-profiling="false" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669242 2581 flags.go:64] FLAG: --cpu-cfs-quota="true" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669245 2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669248 2581 flags.go:64] FLAG: --cpu-manager-policy="none" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669251 2581 flags.go:64] FLAG: --cpu-manager-policy-options="" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669254 2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669257 2581 flags.go:64] FLAG: --enable-controller-attach-detach="true" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669260 2581 flags.go:64] FLAG: --enable-debugging-handlers="true" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669263 2581 flags.go:64] FLAG: --enable-load-reader="false" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669266 2581 flags.go:64] FLAG: --enable-server="true" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669269 2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669273 2581 flags.go:64] FLAG: --event-burst="100" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669276 2581 flags.go:64] FLAG: --event-qps="50" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669278 2581 flags.go:64] FLAG: --event-storage-age-limit="default=0" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669282 2581 flags.go:64] FLAG: --event-storage-event-limit="default=0" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669285 2581 flags.go:64] FLAG: --eviction-hard="" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669288 2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Jul 03 11:12:16.673700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669291 2581 flags.go:64] FLAG: --eviction-minimum-reclaim="" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669294 2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669297 2581 flags.go:64] FLAG: --eviction-soft="" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669300 2581 flags.go:64] FLAG: --eviction-soft-grace-period="" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669303 2581 flags.go:64] FLAG: --exit-on-lock-contention="false" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669306 2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669309 2581 flags.go:64] FLAG: --experimental-mounter-path="" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669312 2581 flags.go:64] FLAG: --fail-cgroupv1="false" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669316 2581 flags.go:64] FLAG: --fail-swap-on="true" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669319 2581 flags.go:64] FLAG: --feature-gates="" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669323 2581 flags.go:64] FLAG: --file-check-frequency="20s" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669326 2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669329 2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669332 2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669335 2581 flags.go:64] FLAG: --healthz-port="10248" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669338 2581 flags.go:64] FLAG: --help="false" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669340 2581 flags.go:64] FLAG: --hostname-override="ip-10-0-141-89.ec2.internal" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669343 2581 flags.go:64] FLAG: --housekeeping-interval="10s" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669346 2581 flags.go:64] FLAG: --http-check-frequency="20s" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669349 2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669352 2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669356 2581 flags.go:64] FLAG: --image-gc-high-threshold="85" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669358 2581 flags.go:64] FLAG: --image-gc-low-threshold="80" Jul 03 11:12:16.674271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669361 2581 flags.go:64] FLAG: --image-service-endpoint="" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669364 2581 flags.go:64] FLAG: --kernel-memcg-notification="false" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669368 2581 flags.go:64] FLAG: --kube-api-burst="100" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669370 2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669373 2581 flags.go:64] FLAG: --kube-api-qps="50" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669376 2581 flags.go:64] FLAG: --kube-reserved="" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669379 2581 flags.go:64] FLAG: --kube-reserved-cgroup="" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669382 2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669385 2581 flags.go:64] FLAG: --kubelet-cgroups="" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669387 2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669390 2581 flags.go:64] FLAG: --lock-file="" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669393 2581 flags.go:64] FLAG: --log-cadvisor-usage="false" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669396 2581 flags.go:64] FLAG: --log-flush-frequency="5s" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669398 2581 flags.go:64] FLAG: --log-json-info-buffer-size="0" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669403 2581 flags.go:64] FLAG: --log-json-split-stream="false" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669407 2581 flags.go:64] FLAG: --log-text-info-buffer-size="0" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669410 2581 flags.go:64] FLAG: --log-text-split-stream="false" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669413 2581 flags.go:64] FLAG: --logging-format="text" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669416 2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669419 2581 flags.go:64] FLAG: --make-iptables-util-chains="true" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669422 2581 flags.go:64] FLAG: --manifest-url="" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669425 2581 flags.go:64] FLAG: --manifest-url-header="" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669429 2581 flags.go:64] FLAG: --max-housekeeping-interval="15s" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669432 2581 flags.go:64] FLAG: --max-open-files="1000000" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669436 2581 flags.go:64] FLAG: --max-pods="110" Jul 03 11:12:16.674962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669439 2581 flags.go:64] FLAG: --maximum-dead-containers="-1" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669442 2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669445 2581 flags.go:64] FLAG: --memory-manager-policy="None" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669447 2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669450 2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669453 2581 flags.go:64] FLAG: --node-ip="0.0.0.0" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669456 2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669463 2581 flags.go:64] FLAG: --node-status-max-images="50" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669466 2581 flags.go:64] FLAG: --node-status-update-frequency="10s" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669469 2581 flags.go:64] FLAG: --oom-score-adj="-999" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669472 2581 flags.go:64] FLAG: --pod-cidr="" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669475 2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:ce93f6b4197b9c62a26a3131f8057ffe9c9379c5079a4f62e5919b75151394e5" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669481 2581 flags.go:64] FLAG: --pod-manifest-path="" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669483 2581 flags.go:64] FLAG: --pod-max-pids="-1" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669486 2581 flags.go:64] FLAG: --pods-per-core="0" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669489 2581 flags.go:64] FLAG: --port="10250" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669492 2581 flags.go:64] FLAG: --protect-kernel-defaults="false" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669495 2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08f8f6854c5315474" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669498 2581 flags.go:64] FLAG: --qos-reserved="" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669501 2581 flags.go:64] FLAG: --read-only-port="10255" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669504 2581 flags.go:64] FLAG: --register-node="true" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669506 2581 flags.go:64] FLAG: --register-with-taints="" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669510 2581 flags.go:64] FLAG: --registry-burst="10" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669513 2581 flags.go:64] FLAG: --registry-qps="5" Jul 03 11:12:16.675568 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669517 2581 flags.go:64] FLAG: --reserved-cpus="" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669520 2581 flags.go:64] FLAG: --reserved-memory="" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669523 2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669527 2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669529 2581 flags.go:64] FLAG: --rotate-certificates="false" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669532 2581 flags.go:64] FLAG: --rotate-server-certificates="false" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669535 2581 flags.go:64] FLAG: --runonce="false" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669538 2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669540 2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669543 2581 flags.go:64] FLAG: --seccomp-default="false" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669546 2581 flags.go:64] FLAG: --serialize-image-pulls="true" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669549 2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669553 2581 flags.go:64] FLAG: --storage-driver-db="cadvisor" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669556 2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669559 2581 flags.go:64] FLAG: --storage-driver-password="root" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669562 2581 flags.go:64] FLAG: --storage-driver-secure="false" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669564 2581 flags.go:64] FLAG: --storage-driver-table="stats" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669567 2581 flags.go:64] FLAG: --storage-driver-user="root" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669570 2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669573 2581 flags.go:64] FLAG: --sync-frequency="1m0s" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669575 2581 flags.go:64] FLAG: --system-cgroups="" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669578 2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669583 2581 flags.go:64] FLAG: --system-reserved-cgroup="" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669586 2581 flags.go:64] FLAG: --tls-cert-file="" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669589 2581 flags.go:64] FLAG: --tls-cipher-suites="[]" Jul 03 11:12:16.676192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669593 2581 flags.go:64] FLAG: --tls-min-version="" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669596 2581 flags.go:64] FLAG: --tls-private-key-file="" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669598 2581 flags.go:64] FLAG: --topology-manager-policy="none" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669601 2581 flags.go:64] FLAG: --topology-manager-policy-options="" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669604 2581 flags.go:64] FLAG: --topology-manager-scope="container" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669607 2581 flags.go:64] FLAG: --v="2" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669611 2581 flags.go:64] FLAG: --version="false" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669617 2581 flags.go:64] FLAG: --vmodule="" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669621 2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.669624 2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669785 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669790 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669793 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669796 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669799 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669802 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669804 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669807 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669810 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669814 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669817 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669820 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 11:12:16.676832 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669822 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669825 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669827 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669830 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669833 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669837 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669840 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669843 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669846 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669849 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669851 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669854 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669856 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669859 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669863 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669866 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669869 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669873 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669877 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 11:12:16.677365 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669880 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669882 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669885 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669887 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669890 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669892 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669895 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669897 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669899 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669902 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669905 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669908 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669910 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669913 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669915 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669917 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669920 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669923 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669925 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669928 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 11:12:16.677852 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669930 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669932 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669935 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669937 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669940 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669942 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669945 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669947 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669950 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669952 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669957 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669959 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669962 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669965 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669967 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669970 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669972 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669974 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669977 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669980 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669982 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 11:12:16.678379 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669984 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669988 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669992 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669996 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.669998 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670001 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670003 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670006 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670008 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670011 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670013 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670016 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670018 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670021 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670023 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670026 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670029 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670032 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670034 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 11:12:16.678916 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670037 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670040 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670044 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670046 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670049 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670053 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670055 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.670058 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.670585 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.677903 2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.8" Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.677919 2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.677937 2581 watchdog_linux.go:95] "Systemd watchdog is not enabled" Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.677946 2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678008 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678012 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 11:12:16.679380 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678016 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678019 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678022 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678024 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678028 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678031 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678034 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678037 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678040 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678043 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678046 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678048 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678051 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678053 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678056 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678058 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678061 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678063 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678065 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678068 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 11:12:16.679782 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678071 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678075 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678079 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678081 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678084 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678087 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678090 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678099 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678102 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678104 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678107 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678110 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678113 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678116 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678118 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678121 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678124 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678126 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678129 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 11:12:16.680263 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678133 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678136 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678139 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678141 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678145 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678147 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678150 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678152 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678154 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678157 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678159 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678161 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678164 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678166 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678169 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678171 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678173 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678176 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678179 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678181 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 11:12:16.680763 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678183 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678191 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678194 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678196 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678199 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678201 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678204 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678207 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678209 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678212 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678214 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678217 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678219 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678221 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678224 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678226 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678229 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678231 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678234 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678236 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 11:12:16.681250 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678238 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678240 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678243 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678245 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678248 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678250 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678252 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678254 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678257 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678259 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678262 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678264 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678267 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678271 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678279 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678282 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678285 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 11:12:16.681798 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678287 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.678292 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678425 2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678432 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678435 2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678438 2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678440 2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678443 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678445 2581 feature_gate.go:328] unrecognized feature gate: SignatureStores Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678449 2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678453 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678455 2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678458 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678460 2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement Jul 03 11:12:16.682240 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678463 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678465 2581 feature_gate.go:328] unrecognized feature gate: EventTTL Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678468 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678470 2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678473 2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678475 2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678478 2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678480 2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678483 2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678485 2581 feature_gate.go:328] unrecognized feature gate: Example2 Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678487 2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678490 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678492 2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678495 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678497 2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678499 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678507 2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678510 2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678514 2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release. Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678517 2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace Jul 03 11:12:16.682602 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678520 2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678523 2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678526 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678529 2581 feature_gate.go:328] unrecognized feature gate: PinnedImages Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678532 2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678534 2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678537 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678540 2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678542 2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678545 2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678547 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678550 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678553 2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678555 2581 feature_gate.go:328] unrecognized feature gate: Example Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678558 2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678560 2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678563 2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678565 2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678568 2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678570 2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform Jul 03 11:12:16.683201 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678572 2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678574 2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678577 2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678580 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678582 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678584 2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678587 2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678589 2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678591 2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678600 2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678603 2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678605 2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678608 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678610 2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678613 2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678615 2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678618 2581 feature_gate.go:328] unrecognized feature gate: DualReplica Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678620 2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678623 2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678625 2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure Jul 03 11:12:16.683724 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678628 2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678630 2581 feature_gate.go:328] unrecognized feature gate: OVNObservability Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678632 2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678635 2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678652 2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678655 2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678657 2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678660 2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678662 2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678665 2581 feature_gate.go:328] unrecognized feature gate: OSStreams Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678667 2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678669 2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678672 2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678674 2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678677 2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678680 2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678682 2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678685 2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678687 2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig Jul 03 11:12:16.684221 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678690 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678692 2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678695 2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678703 2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678705 2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678708 2581 feature_gate.go:328] unrecognized feature gate: NewOLM Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678710 2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:16.678713 2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.678717 2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]} Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.678853 2581 server.go:962] "Client rotation is on, will bootstrap in background" Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.680768 2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.682296 2581 server.go:1019] "Starting client certificate rotation" Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.682410 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 03 11:12:16.684708 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.682446 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Jul 03 11:12:16.685150 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.685134 2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 03 11:12:16.686916 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.686892 2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Jul 03 11:12:16.698540 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.698517 2581 log.go:25] "Validated CRI v1 runtime API" Jul 03 11:12:16.700655 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.700624 2581 log.go:25] "Validated CRI v1 image API" Jul 03 11:12:16.703151 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.703134 2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Jul 03 11:12:16.706438 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.706418 2581 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 7d529ba6-1368-4ac1-9b0d-8d054166af79:/dev/nvme0n1p3 d464ce32-5a6b-4995-93b3-4e7d3c25d2d6:/dev/nvme0n1p4] Jul 03 11:12:16.706489 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.706439 2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}] Jul 03 11:12:16.713030 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.712900 2581 manager.go:216] Machine: {Timestamp:2026-07-03 11:12:16.711454955 +0000 UTC m=+0.379923381 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098047 MemoryCapacity:32812179456 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21302f20fd2f632499962a56439688 SystemUUID:ec21302f-20fd-2f63-2499-962a56439688 BootID:2844ef13-3534-4cef-b80d-20c52bdfadb2 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406089728 Type:vfs Inodes:4005393 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562439168 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:68:9a:31:9d:af Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:68:9a:31:9d:af Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:94:1d:8f:c5:0a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812179456 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Jul 03 11:12:16.713030 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.713015 2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Jul 03 11:12:16.713169 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.713101 2581 manager.go:232] Version: {KernelVersion:5.14.0-570.125.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260630-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Jul 03 11:12:16.715386 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.715367 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 03 11:12:16.717166 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.717136 2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Jul 03 11:12:16.717320 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.717170 2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-89.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Jul 03 11:12:16.717975 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.717965 2581 topology_manager.go:138] "Creating topology manager with none policy" Jul 03 11:12:16.718008 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.717978 2581 container_manager_linux.go:309] "Creating device plugin manager" Jul 03 11:12:16.718008 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.717985 2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 03 11:12:16.718008 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.717999 2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Jul 03 11:12:16.719546 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.719536 2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager" Jul 03 11:12:16.722084 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.722072 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 03 11:12:16.722380 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.722369 2581 server.go:1240] "Using root directory" path="/var/lib/kubelet" Jul 03 11:12:16.722885 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.722872 2581 kubelet.go:486] "Attempting to sync node with API server" Jul 03 11:12:16.722947 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.722894 2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests" Jul 03 11:12:16.722947 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.722915 2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Jul 03 11:12:16.722947 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.722924 2581 kubelet.go:398] "Adding apiserver pod source" Jul 03 11:12:16.722947 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.722933 2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Jul 03 11:12:16.723350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.723339 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 03 11:12:16.723417 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.723358 2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Jul 03 11:12:16.725812 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.725793 2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.8-3.rhaos4.21.git91b4b0b.el9" apiVersion="v1" Jul 03 11:12:16.727508 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.727495 2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Jul 03 11:12:16.727570 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.727525 2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Jul 03 11:12:16.728609 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728596 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Jul 03 11:12:16.728675 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728614 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Jul 03 11:12:16.728675 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728620 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Jul 03 11:12:16.728675 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728630 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Jul 03 11:12:16.728675 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728660 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Jul 03 11:12:16.728675 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728667 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Jul 03 11:12:16.728675 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728676 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Jul 03 11:12:16.728831 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728681 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Jul 03 11:12:16.728831 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728689 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Jul 03 11:12:16.728831 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728695 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Jul 03 11:12:16.728831 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728704 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Jul 03 11:12:16.728831 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.728709 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Jul 03 11:12:16.729358 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.729348 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Jul 03 11:12:16.729358 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.729358 2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Jul 03 11:12:16.732688 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.732668 2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-89.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Jul 03 11:12:16.732767 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.732672 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" Jul 03 11:12:16.732888 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.732854 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" Jul 03 11:12:16.733061 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.733048 2581 server.go:1268] "Started kubelet" Jul 03 11:12:16.733230 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.733198 2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250 Jul 03 11:12:16.733394 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.733190 2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Jul 03 11:12:16.733453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.733426 2581 server_v1.go:49] "podresources" method="list" useActivePods=true Jul 03 11:12:16.734083 ip-10-0-141-89 systemd[1]: Started Kubernetes Kubelet. Jul 03 11:12:16.734995 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.734983 2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Jul 03 11:12:16.739971 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.739945 2581 server.go:310] "Adding debug handlers to kubelet server" Jul 03 11:12:16.741088 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.741062 2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Jul 03 11:12:16.743728 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.742779 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-89.ec2.internal.18bec3356e179929 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-89.ec2.internal,UID:ip-10-0-141-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-89.ec2.internal,},FirstTimestamp:2026-07-03 11:12:16.733018409 +0000 UTC m=+0.401486833,LastTimestamp:2026-07-03 11:12:16.733018409 +0000 UTC m=+0.401486833,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-89.ec2.internal,}" Jul 03 11:12:16.745052 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.745031 2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer" Jul 03 11:12:16.745247 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.745144 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:16.745354 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.745336 2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Jul 03 11:12:16.745728 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.745705 2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Jul 03 11:12:16.745820 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.745782 2581 volume_manager.go:311] "The desired_state_of_world populator starts" Jul 03 11:12:16.745820 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.745795 2581 volume_manager.go:313] "Starting Kubelet Volume Manager" Jul 03 11:12:16.745917 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.745890 2581 reconstruct.go:97] "Volume reconstruction finished" Jul 03 11:12:16.745917 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.745902 2581 reconciler.go:29] "Reconciler: start to sync state" Jul 03 11:12:16.747243 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.747222 2581 factory.go:153] Registering CRI-O factory Jul 03 11:12:16.747243 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.747238 2581 factory.go:223] Registration of the crio container factory successfully Jul 03 11:12:16.747376 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.747288 2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Jul 03 11:12:16.747376 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.747298 2581 factory.go:55] Registering systemd factory Jul 03 11:12:16.747376 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.747304 2581 factory.go:223] Registration of the systemd container factory successfully Jul 03 11:12:16.747376 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.747322 2581 factory.go:103] Registering Raw factory Jul 03 11:12:16.747376 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.747335 2581 manager.go:1196] Started watching for new ooms in manager Jul 03 11:12:16.748247 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.748232 2581 manager.go:349] Starting recovery of all containers Jul 03 11:12:16.750347 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.750318 2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" Jul 03 11:12:16.750450 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.750420 2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-89.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Jul 03 11:12:16.759915 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.759765 2581 manager.go:354] Recovery completed Jul 03 11:12:16.764441 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.764426 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 11:12:16.769855 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.769835 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientMemory" Jul 03 11:12:16.769943 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.769875 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 11:12:16.769943 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.769891 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientPID" Jul 03 11:12:16.770465 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.770452 2581 cpu_manager.go:222] "Starting CPU manager" policy="none" Jul 03 11:12:16.770465 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.770464 2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s" Jul 03 11:12:16.770565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.770479 2581 state_mem.go:36] "Initialized new in-memory state store" Jul 03 11:12:16.770565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.770529 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 11:12:16.771558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.771540 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientMemory" Jul 03 11:12:16.771627 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.771577 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 11:12:16.771627 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.771592 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientPID" Jul 03 11:12:16.772058 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.771958 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-89.ec2.internal.18bec3357049b78f default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-89.ec2.internal,UID:ip-10-0-141-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-89.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-89.ec2.internal,},FirstTimestamp:2026-07-03 11:12:16.769857423 +0000 UTC m=+0.438325850,LastTimestamp:2026-07-03 11:12:16.769857423 +0000 UTC m=+0.438325850,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-89.ec2.internal,}" Jul 03 11:12:16.772150 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.772111 2581 policy_none.go:49] "None policy: Start" Jul 03 11:12:16.772150 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.772124 2581 memory_manager.go:187] "Starting memorymanager" policy="None" Jul 03 11:12:16.772150 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.772137 2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Jul 03 11:12:16.773048 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.773038 2581 policy_none.go:47] "Start" Jul 03 11:12:16.773971 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.773954 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-85pgb" Jul 03 11:12:16.782524 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.782456 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-89.ec2.internal.18bec335704a19e0 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-89.ec2.internal,UID:ip-10-0-141-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-89.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-89.ec2.internal,},FirstTimestamp:2026-07-03 11:12:16.769882592 +0000 UTC m=+0.438351018,LastTimestamp:2026-07-03 11:12:16.769882592 +0000 UTC m=+0.438351018,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-89.ec2.internal,}" Jul 03 11:12:16.787195 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.787173 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-85pgb" Jul 03 11:12:16.797072 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.797001 2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-89.ec2.internal.18bec335704a537c default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-89.ec2.internal,UID:ip-10-0-141-89.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-141-89.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-141-89.ec2.internal,},FirstTimestamp:2026-07-03 11:12:16.76989734 +0000 UTC m=+0.438365767,LastTimestamp:2026-07-03 11:12:16.76989734 +0000 UTC m=+0.438365767,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-89.ec2.internal,}" Jul 03 11:12:16.812266 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.812248 2581 manager.go:337] "Starting Device Plugin manager" Jul 03 11:12:16.812384 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.812286 2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Jul 03 11:12:16.812384 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.812297 2581 server.go:87] "Starting device plugin registration server" Jul 03 11:12:16.812618 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.812604 2581 eviction_manager.go:189] "Eviction manager: starting control loop" Jul 03 11:12:16.812698 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.812618 2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Jul 03 11:12:16.813300 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.813278 2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Jul 03 11:12:16.813423 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.813379 2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts" Jul 03 11:12:16.813423 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.813389 2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager" Jul 03 11:12:16.813568 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.813417 2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Jul 03 11:12:16.813568 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.813474 2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:16.880570 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.880527 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Jul 03 11:12:16.881933 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.881918 2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Jul 03 11:12:16.882000 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.881967 2581 status_manager.go:244] "Starting to sync pod status with apiserver" Jul 03 11:12:16.882000 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.881996 2581 kubelet.go:2448] "Starting kubelet main sync loop" Jul 03 11:12:16.882081 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.882037 2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Jul 03 11:12:16.884611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.884586 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 11:12:16.913700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.913598 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 11:12:16.914969 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.914941 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientMemory" Jul 03 11:12:16.914969 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.914974 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 11:12:16.915177 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.914990 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientPID" Jul 03 11:12:16.915177 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.915048 2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-89.ec2.internal" Jul 03 11:12:16.926158 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.926141 2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-89.ec2.internal" Jul 03 11:12:16.926234 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.926167 2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-89.ec2.internal\": node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:16.952411 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:16.952367 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:16.983051 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.983016 2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal"] Jul 03 11:12:16.983124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.983104 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 11:12:16.984138 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.984120 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientMemory" Jul 03 11:12:16.984225 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.984153 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 11:12:16.984225 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.984165 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientPID" Jul 03 11:12:16.985397 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.985384 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 11:12:16.985541 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.985525 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:16.985576 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.985559 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 11:12:16.986205 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.986189 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientMemory" Jul 03 11:12:16.986277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.986216 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 11:12:16.986277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.986218 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientMemory" Jul 03 11:12:16.986277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.986227 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientPID" Jul 03 11:12:16.986277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.986251 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 11:12:16.986277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.986266 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientPID" Jul 03 11:12:16.987193 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.987179 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" Jul 03 11:12:16.987230 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.987210 2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach" Jul 03 11:12:16.987868 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.987852 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientMemory" Jul 03 11:12:16.987959 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.987883 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasNoDiskPressure" Jul 03 11:12:16.987959 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:16.987896 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeHasSufficientPID" Jul 03 11:12:17.010474 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:17.010451 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-89.ec2.internal\" not found" node="ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.014935 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:17.014916 2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-89.ec2.internal\" not found" node="ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.047422 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.047381 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/016eda45f6076f69f860c98ef70bfd7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal\" (UID: \"016eda45f6076f69f860c98ef70bfd7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.047422 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.047421 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/016eda45f6076f69f860c98ef70bfd7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal\" (UID: \"016eda45f6076f69f860c98ef70bfd7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.047624 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.047446 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36c40a12ef0a62b4aa40697a8c2cbf13-config\") pod \"kube-apiserver-proxy-ip-10-0-141-89.ec2.internal\" (UID: \"36c40a12ef0a62b4aa40697a8c2cbf13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.053498 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:17.053476 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:17.147896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.147863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/016eda45f6076f69f860c98ef70bfd7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal\" (UID: \"016eda45f6076f69f860c98ef70bfd7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.147896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.147893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/016eda45f6076f69f860c98ef70bfd7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal\" (UID: \"016eda45f6076f69f860c98ef70bfd7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.148090 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.147917 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36c40a12ef0a62b4aa40697a8c2cbf13-config\") pod \"kube-apiserver-proxy-ip-10-0-141-89.ec2.internal\" (UID: \"36c40a12ef0a62b4aa40697a8c2cbf13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.148090 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.147977 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/016eda45f6076f69f860c98ef70bfd7f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal\" (UID: \"016eda45f6076f69f860c98ef70bfd7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.148090 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.148041 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/016eda45f6076f69f860c98ef70bfd7f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal\" (UID: \"016eda45f6076f69f860c98ef70bfd7f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.148090 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.148087 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/36c40a12ef0a62b4aa40697a8c2cbf13-config\") pod \"kube-apiserver-proxy-ip-10-0-141-89.ec2.internal\" (UID: \"36c40a12ef0a62b4aa40697a8c2cbf13\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.153978 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:17.153958 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:17.254931 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:17.254848 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:17.314057 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.314028 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:17.318878 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:17.318857 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" Jul 03 11:12:18.461297 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:18.461260 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:18.561885 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:18.561842 2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-89.ec2.internal\" not found" Jul 03 11:12:18.652322 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.652259 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 11:12:18.751154 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.751109 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" Jul 03 11:12:18.766853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.766826 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 03 11:12:18.768935 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.768920 2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" Jul 03 11:12:18.786691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.786662 2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Jul 03 11:12:18.787748 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.787731 2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Jul 03 11:12:18.787872 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.787851 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 03 11:12:18.787926 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.787858 2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received" Jul 03 11:12:18.829086 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.829056 2581 apiserver.go:52] "Watching apiserver" Jul 03 11:12:18.851854 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.851817 2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Jul 03 11:12:18.860085 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.860059 2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Jul 03 11:12:18.860401 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.860382 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal","openshift-cluster-node-tuning-operator/tuned-w4vkh","openshift-image-registry/node-ca-mc48b","openshift-network-operator/iptables-alerter-ft7vq","openshift-ovn-kubernetes/ovnkube-node-47drl","kube-system/global-pull-secret-syncer-q2wrq","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5","openshift-dns/node-resolver-8d5tj","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal","openshift-multus/multus-additional-cni-plugins-8tn46","openshift-multus/multus-q5wgx","openshift-multus/network-metrics-daemon-dqt7b","openshift-network-diagnostics/network-check-target-76w8c","kube-system/konnectivity-agent-wkzwc"] Jul 03 11:12:18.861993 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.861976 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.863095 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.863079 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:18.864155 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.864140 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:18.867711 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.867303 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.868530 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.868511 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:18.868609 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:18.868587 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:18.869604 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.869566 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:18.870703 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.870684 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:18.871858 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.871844 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.872871 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.872854 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Jul 03 11:12:18.873113 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.873094 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.873664 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.873649 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-qtl57\"" Jul 03 11:12:18.875273 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875255 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Jul 03 11:12:18.875363 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875330 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:18.875363 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875356 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Jul 03 11:12:18.875508 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875257 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Jul 03 11:12:18.875508 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:18.875405 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:18.875508 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875488 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Jul 03 11:12:18.875653 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875517 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Jul 03 11:12:18.875653 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875554 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Jul 03 11:12:18.875653 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875617 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Jul 03 11:12:18.875791 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.875727 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Jul 03 11:12:18.876810 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876568 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Jul 03 11:12:18.876810 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876583 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Jul 03 11:12:18.876810 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876586 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Jul 03 11:12:18.876810 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876704 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Jul 03 11:12:18.876810 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876714 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Jul 03 11:12:18.876810 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876718 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Jul 03 11:12:18.877075 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876880 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:18.877075 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876923 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Jul 03 11:12:18.877075 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.876954 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Jul 03 11:12:18.877075 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:18.876992 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:18.877608 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.877587 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-z4ppc\"" Jul 03 11:12:18.877715 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.877610 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-v74bj\"" Jul 03 11:12:18.877715 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.877649 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-925q4\"" Jul 03 11:12:18.877715 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.877585 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Jul 03 11:12:18.877715 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.877674 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Jul 03 11:12:18.877715 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.877710 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m2dt2\"" Jul 03 11:12:18.877715 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.877648 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-57wjb\"" Jul 03 11:12:18.878025 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.877652 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Jul 03 11:12:18.878131 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.878118 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:18.891009 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.890980 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Jul 03 11:12:18.891009 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.890985 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Jul 03 11:12:18.891558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.891543 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Jul 03 11:12:18.891558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.891552 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Jul 03 11:12:18.896305 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.896277 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Jul 03 11:12:18.897268 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.897255 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Jul 03 11:12:18.897268 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.897263 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-76nlv\"" Jul 03 11:12:18.897379 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.897257 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Jul 03 11:12:18.898065 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.898050 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-znzw9\"" Jul 03 11:12:18.898784 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.898769 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l6gcs\"" Jul 03 11:12:18.901490 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.901458 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-02 11:07:17 +0000 UTC" deadline="2028-03-06 08:39:16.622255731 +0000 UTC" Jul 03 11:12:18.901490 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.901488 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14685h26m57.720770687s" Jul 03 11:12:18.901619 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.901513 2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Jul 03 11:12:18.953004 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.952986 2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Jul 03 11:12:18.957514 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957489 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-cni-bin\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.957630 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957518 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-cni-multus\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.957630 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957534 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-k8s-cni-cncf-io\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.957630 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhlj\" (UniqueName: \"kubernetes.io/projected/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-kube-api-access-ckhlj\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.957630 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957576 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-ovn\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.957630 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957591 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-cni-bin\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.957630 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957621 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a6b54da-ff3a-4ad1-868b-1f06684af304-hosts-file\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957646 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6b54da-ff3a-4ad1-868b-1f06684af304-tmp-dir\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957667 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-etc-kubernetes\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957683 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957704 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64twv\" (UniqueName: \"kubernetes.io/projected/5dc69354-77c5-484c-b655-7ff694e5e9d9-kube-api-access-64twv\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957722 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mz7qs\" (UniqueName: \"kubernetes.io/projected/b4cfd292-27df-444a-b0af-43067d41ee34-kube-api-access-mz7qs\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-device-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t478f\" (UniqueName: \"kubernetes.io/projected/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-kube-api-access-t478f\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957774 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-cnibin\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957827 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:18.957870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-lib-modules\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957877 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/629b92be-f77d-4067-8732-cea6d1e6c2dd-tmp\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957893 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxjqh\" (UniqueName: \"kubernetes.io/projected/6e5067b3-011f-4a3b-a128-86ce147a1c69-kube-api-access-qxjqh\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957911 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-var-lib-kubelet\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957934 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4cfd292-27df-444a-b0af-43067d41ee34-serviceca\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957959 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-conf-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.957988 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-var-lib-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958003 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-host\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-kubelet\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958072 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958105 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:18.958277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958131 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xwkz\" (UniqueName: \"kubernetes.io/projected/629b92be-f77d-4067-8732-cea6d1e6c2dd-kube-api-access-7xwkz\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.958841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958155 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-cni-binary-copy\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.958841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958538 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-socket-dir-parent\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.958841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958580 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-etc-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.958841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958634 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovn-node-metrics-cert\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.958841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958688 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-system-cni-dir\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.958841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958730 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-kubernetes\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.958841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-cnibin\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.959136 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-kubelet\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.959136 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.959136 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958920 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-sys\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.959136 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958958 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxg7m\" (UniqueName: \"kubernetes.io/projected/1a6b54da-ff3a-4ad1-868b-1f06684af304-kube-api-access-sxg7m\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:18.959136 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.958986 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-cni-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.959136 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959017 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-systemd-units\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.959136 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959048 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.959136 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwwpv\" (UniqueName: \"kubernetes.io/projected/e6925168-d48c-405d-8a6c-30384df3214c-kube-api-access-vwwpv\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:18.959486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959151 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-run\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.959486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959179 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-os-release\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.959486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959226 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-netns\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.959486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959253 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-slash\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.959486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959283 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-run-netns\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.959486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959307 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b814191-d67d-47b0-8c3f-af739872704b-kubelet-config\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:18.959486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b814191-d67d-47b0-8c3f-af739872704b-dbus\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:18.959777 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/051916bd-975d-49bf-810c-768ed087aa68-konnectivity-ca\") pod \"konnectivity-agent-wkzwc\" (UID: \"051916bd-975d-49bf-810c-768ed087aa68\") " pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:18.959777 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959662 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-socket-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:18.959777 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959694 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-registration-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:18.959777 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959725 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-node-log\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.959777 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959751 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-cni-netd\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.959998 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959778 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/051916bd-975d-49bf-810c-768ed087aa68-agent-certs\") pod \"konnectivity-agent-wkzwc\" (UID: \"051916bd-975d-49bf-810c-768ed087aa68\") " pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:18.959998 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959805 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-systemd\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.959998 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959832 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-tuned\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.959998 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959855 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d0b8313-5882-4bcb-8192-91a980250a70-host-slash\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:18.959998 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959881 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-system-cni-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.959998 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.959914 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-daemon-config\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.960261 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960009 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-systemd\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.960261 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960058 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-env-overrides\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.960261 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960083 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-os-release\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.960261 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960127 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d0b8313-5882-4bcb-8192-91a980250a70-iptables-alerter-script\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:18.960261 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960154 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k92n\" (UniqueName: \"kubernetes.io/projected/6d0b8313-5882-4bcb-8192-91a980250a70-kube-api-access-9k92n\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:18.960261 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-sys-fs\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:18.960613 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960263 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-log-socket\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.960613 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960299 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovnkube-config\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.960613 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960378 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-modprobe-d\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.960613 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960475 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysctl-conf\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.960613 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-hostroot\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.960613 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960552 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-multus-certs\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:18.960613 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960585 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4cfd292-27df-444a-b0af-43067d41ee34-host\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:18.960613 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960611 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovnkube-script-lib\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:18.960983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960660 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.960983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960696 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:18.960983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960727 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:18.960983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960753 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:18.960983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960783 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysconfig\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.960983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.960813 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysctl-d\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:18.961741 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.961719 2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-r7hw9" Jul 03 11:12:18.970256 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:18.970238 2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-r7hw9" Jul 03 11:12:19.009760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.009735 2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 11:12:19.061801 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061772 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.061934 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061808 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysconfig\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.061934 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061826 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysctl-d\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.061934 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-cni-bin\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.061934 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061860 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-cni-multus\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.061934 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061918 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-cni-multus\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061918 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysconfig\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061940 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-etc-selinux\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061967 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-k8s-cni-cncf-io\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysctl-d\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-k8s-cni-cncf-io\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062031 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhlj\" (UniqueName: \"kubernetes.io/projected/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-kube-api-access-ckhlj\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062048 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-ovn\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062063 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-cni-bin\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062078 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a6b54da-ff3a-4ad1-868b-1f06684af304-hosts-file\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062093 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6b54da-ff3a-4ad1-868b-1f06684af304-tmp-dir\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.061918 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-cni-bin\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062112 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-etc-kubernetes\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062114 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-ovn\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.062124 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062127 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-cni-bin\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062136 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64twv\" (UniqueName: \"kubernetes.io/projected/5dc69354-77c5-484c-b655-7ff694e5e9d9-kube-api-access-64twv\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062164 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/1a6b54da-ff3a-4ad1-868b-1f06684af304-hosts-file\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062189 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-etc-kubernetes\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062223 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062198 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mz7qs\" (UniqueName: \"kubernetes.io/projected/b4cfd292-27df-444a-b0af-43067d41ee34-kube-api-access-mz7qs\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-device-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062336 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t478f\" (UniqueName: \"kubernetes.io/projected/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-kube-api-access-t478f\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-cnibin\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062391 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a6b54da-ff3a-4ad1-868b-1f06684af304-tmp-dir\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062394 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-device-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062464 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-cnibin\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062459 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-lib-modules\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062606 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-kubelet-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.062790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062635 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/629b92be-f77d-4067-8732-cea6d1e6c2dd-tmp\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062665 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-lib-modules\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062699 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxjqh\" (UniqueName: \"kubernetes.io/projected/6e5067b3-011f-4a3b-a128-86ce147a1c69-kube-api-access-qxjqh\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062758 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-var-lib-kubelet\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062784 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4cfd292-27df-444a-b0af-43067d41ee34-serviceca\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062810 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-conf-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062846 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-var-lib-kubelet\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062861 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-conf-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062872 2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-var-lib-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062910 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062945 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-host\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062951 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-var-lib-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.062971 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-kubelet\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063001 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-host\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063004 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063035 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-var-lib-kubelet\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063035 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:19.063589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xwkz\" (UniqueName: \"kubernetes.io/projected/629b92be-f77d-4067-8732-cea6d1e6c2dd-kube-api-access-7xwkz\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063109 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-cni-binary-copy\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063075 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-run-ovn-kubernetes\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063137 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-socket-dir-parent\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.063154 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063188 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b4cfd292-27df-444a-b0af-43067d41ee34-serviceca\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.063257 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret podName:0b814191-d67d-47b0-8c3f-af739872704b nodeName:}" failed. No retries permitted until 2026-07-03 11:12:19.563237371 +0000 UTC m=+2.125912882 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret") pod "global-pull-secret-syncer-q2wrq" (UID: "0b814191-d67d-47b0-8c3f-af739872704b") : object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-etc-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063304 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovn-node-metrics-cert\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063326 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-system-cni-dir\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063342 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-etc-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063384 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-system-cni-dir\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-kubernetes\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063397 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-kubernetes\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063421 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-cnibin\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-kubelet\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063510 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.064453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063471 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-socket-dir-parent\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063517 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063559 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-kubelet\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-sys\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063597 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-cnibin\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063647 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-sys\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063602 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxg7m\" (UniqueName: \"kubernetes.io/projected/1a6b54da-ff3a-4ad1-868b-1f06684af304-kube-api-access-sxg7m\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063690 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-cni-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063714 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-systemd-units\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063740 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063765 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwwpv\" (UniqueName: \"kubernetes.io/projected/e6925168-d48c-405d-8a6c-30384df3214c-kube-api-access-vwwpv\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063791 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-run\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063818 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-os-release\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-netns\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063889 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-slash\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063912 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-run-netns\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063924 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-cni-binary-copy\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063937 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b814191-d67d-47b0-8c3f-af739872704b-kubelet-config\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:19.065491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063975 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b814191-d67d-47b0-8c3f-af739872704b-dbus\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-run\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063981 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-netns\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064000 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/051916bd-975d-49bf-810c-768ed087aa68-konnectivity-ca\") pod \"konnectivity-agent-wkzwc\" (UID: \"051916bd-975d-49bf-810c-768ed087aa68\") " pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.063924 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-cni-binary-copy\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064034 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-os-release\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064046 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-socket-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064059 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/0b814191-d67d-47b0-8c3f-af739872704b-kubelet-config\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-registration-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064076 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-systemd-units\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064106 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-slash\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064108 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-node-log\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064130 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-cni-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064146 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-cni-netd\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064143 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-run-netns\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064131 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-openvswitch\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064203 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-host-cni-netd\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064214 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-node-log\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.066351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064244 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/051916bd-975d-49bf-810c-768ed087aa68-agent-certs\") pod \"konnectivity-agent-wkzwc\" (UID: \"051916bd-975d-49bf-810c-768ed087aa68\") " pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064270 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-registration-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-systemd\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-tuned\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064314 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-systemd\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064326 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/0b814191-d67d-47b0-8c3f-af739872704b-dbus\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064245 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-socket-dir\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064370 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d0b8313-5882-4bcb-8192-91a980250a70-host-slash\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d0b8313-5882-4bcb-8192-91a980250a70-host-slash\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064406 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-system-cni-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064454 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-daemon-config\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064482 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-systemd\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-env-overrides\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064528 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-os-release\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064552 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d0b8313-5882-4bcb-8192-91a980250a70-iptables-alerter-script\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k92n\" (UniqueName: \"kubernetes.io/projected/6d0b8313-5882-4bcb-8192-91a980250a70-kube-api-access-9k92n\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-sys-fs\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.067161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064647 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-log-socket\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064671 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovnkube-config\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-modprobe-d\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysctl-conf\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-hostroot\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064768 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-multus-certs\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064809 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4cfd292-27df-444a-b0af-43067d41ee34-host\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064835 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovnkube-script-lib\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064891 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064895 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/051916bd-975d-49bf-810c-768ed087aa68-konnectivity-ca\") pod \"konnectivity-agent-wkzwc\" (UID: \"051916bd-975d-49bf-810c-768ed087aa68\") " pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064920 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.064968 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-log-socket\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065020 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-system-cni-dir\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.065047 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.065099 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs podName:e6925168-d48c-405d-8a6c-30384df3214c nodeName:}" failed. No retries permitted until 2026-07-03 11:12:19.565081188 +0000 UTC m=+2.127756678 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs") pod "network-metrics-daemon-dqt7b" (UID: "e6925168-d48c-405d-8a6c-30384df3214c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065479 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-multus-daemon-config\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.067939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065533 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-run-systemd\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065660 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovnkube-config\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065762 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-modprobe-d\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065874 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-sysctl-conf\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065915 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-env-overrides\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065943 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-hostroot\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.065983 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-host-run-multus-certs\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.066025 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b4cfd292-27df-444a-b0af-43067d41ee34-host\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.066162 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/6e5067b3-011f-4a3b-a128-86ce147a1c69-sys-fs\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.066216 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6d0b8313-5882-4bcb-8192-91a980250a70-iptables-alerter-script\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.066243 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-os-release\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.066350 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/5dc69354-77c5-484c-b655-7ff694e5e9d9-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.066577 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovnkube-script-lib\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.066733 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/5dc69354-77c5-484c-b655-7ff694e5e9d9-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.066779 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/629b92be-f77d-4067-8732-cea6d1e6c2dd-etc-tuned\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.067015 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/629b92be-f77d-4067-8732-cea6d1e6c2dd-tmp\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.067240 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/051916bd-975d-49bf-810c-768ed087aa68-agent-certs\") pod \"konnectivity-agent-wkzwc\" (UID: \"051916bd-975d-49bf-810c-768ed087aa68\") " pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:19.068760 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.067322 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-ovn-node-metrics-cert\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.071362 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.071305 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36c40a12ef0a62b4aa40697a8c2cbf13.slice/crio-936a6b5c57ec0a404c2f5b5165fb850c2dfde9a508ad6b955a4169a960425416 WatchSource:0}: Error finding container 936a6b5c57ec0a404c2f5b5165fb850c2dfde9a508ad6b955a4169a960425416: Status 404 returned error can't find the container with id 936a6b5c57ec0a404c2f5b5165fb850c2dfde9a508ad6b955a4169a960425416 Jul 03 11:12:19.073809 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.073785 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod016eda45f6076f69f860c98ef70bfd7f.slice/crio-d444752ed1e981a80e14242c0017b42fde820b4f38bd0633a564bfc6384e14f7 WatchSource:0}: Error finding container d444752ed1e981a80e14242c0017b42fde820b4f38bd0633a564bfc6384e14f7: Status 404 returned error can't find the container with id d444752ed1e981a80e14242c0017b42fde820b4f38bd0633a564bfc6384e14f7 Jul 03 11:12:19.079109 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.078775 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 11:12:19.079109 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.078795 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 11:12:19.079109 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.078808 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2cnmr for pod openshift-network-diagnostics/network-check-target-76w8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:19.079109 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.078871 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr podName:69bdf6b0-795e-4c72-b8f6-dd577472c2de nodeName:}" failed. No retries permitted until 2026-07-03 11:12:19.578849526 +0000 UTC m=+2.141525017 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2cnmr" (UniqueName: "kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr") pod "network-check-target-76w8c" (UID: "69bdf6b0-795e-4c72-b8f6-dd577472c2de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:19.079852 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.079824 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxjqh\" (UniqueName: \"kubernetes.io/projected/6e5067b3-011f-4a3b-a128-86ce147a1c69-kube-api-access-qxjqh\") pod \"aws-ebs-csi-driver-node-mmmr5\" (UID: \"6e5067b3-011f-4a3b-a128-86ce147a1c69\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.079950 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.079864 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxg7m\" (UniqueName: \"kubernetes.io/projected/1a6b54da-ff3a-4ad1-868b-1f06684af304-kube-api-access-sxg7m\") pod \"node-resolver-8d5tj\" (UID: \"1a6b54da-ff3a-4ad1-868b-1f06684af304\") " pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:19.080013 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.079964 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k92n\" (UniqueName: \"kubernetes.io/projected/6d0b8313-5882-4bcb-8192-91a980250a70-kube-api-access-9k92n\") pod \"iptables-alerter-ft7vq\" (UID: \"6d0b8313-5882-4bcb-8192-91a980250a70\") " pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:19.080013 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.079985 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhlj\" (UniqueName: \"kubernetes.io/projected/60c03a68-858d-4402-98e6-6ce9a2b9a6b2-kube-api-access-ckhlj\") pod \"multus-q5wgx\" (UID: \"60c03a68-858d-4402-98e6-6ce9a2b9a6b2\") " pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.080119 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.080103 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 11:12:19.080770 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.080749 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwwpv\" (UniqueName: \"kubernetes.io/projected/e6925168-d48c-405d-8a6c-30384df3214c-kube-api-access-vwwpv\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:19.081696 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.081678 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mz7qs\" (UniqueName: \"kubernetes.io/projected/b4cfd292-27df-444a-b0af-43067d41ee34-kube-api-access-mz7qs\") pod \"node-ca-mc48b\" (UID: \"b4cfd292-27df-444a-b0af-43067d41ee34\") " pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:19.081769 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.081695 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64twv\" (UniqueName: \"kubernetes.io/projected/5dc69354-77c5-484c-b655-7ff694e5e9d9-kube-api-access-64twv\") pod \"multus-additional-cni-plugins-8tn46\" (UID: \"5dc69354-77c5-484c-b655-7ff694e5e9d9\") " pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.082485 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.082462 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t478f\" (UniqueName: \"kubernetes.io/projected/5a62c8c1-f2e5-4a9a-bda4-530482895a9a-kube-api-access-t478f\") pod \"ovnkube-node-47drl\" (UID: \"5a62c8c1-f2e5-4a9a-bda4-530482895a9a\") " pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.084139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.084123 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xwkz\" (UniqueName: \"kubernetes.io/projected/629b92be-f77d-4067-8732-cea6d1e6c2dd-kube-api-access-7xwkz\") pod \"tuned-w4vkh\" (UID: \"629b92be-f77d-4067-8732-cea6d1e6c2dd\") " pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.188747 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.188661 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" Jul 03 11:12:19.194306 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.194285 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mc48b" Jul 03 11:12:19.197325 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.197296 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod629b92be_f77d_4067_8732_cea6d1e6c2dd.slice/crio-849527f6d8db9b1e2f12cff804af3b8f9e05519d3af31e7b5243e3e0bc4f8b0f WatchSource:0}: Error finding container 849527f6d8db9b1e2f12cff804af3b8f9e05519d3af31e7b5243e3e0bc4f8b0f: Status 404 returned error can't find the container with id 849527f6d8db9b1e2f12cff804af3b8f9e05519d3af31e7b5243e3e0bc4f8b0f Jul 03 11:12:19.202007 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.201983 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4cfd292_27df_444a_b0af_43067d41ee34.slice/crio-b659029b73c978b1a23b31d5cc2fa26849340f9fd16e084eb995e2a292bdf5d2 WatchSource:0}: Error finding container b659029b73c978b1a23b31d5cc2fa26849340f9fd16e084eb995e2a292bdf5d2: Status 404 returned error can't find the container with id b659029b73c978b1a23b31d5cc2fa26849340f9fd16e084eb995e2a292bdf5d2 Jul 03 11:12:19.224253 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.224221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-ft7vq" Jul 03 11:12:19.231836 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.231803 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0b8313_5882_4bcb_8192_91a980250a70.slice/crio-3cb3dceeea80701a7eeda43c4761453bf4e3d7769f998c16ec2f51a4d6a7161c WatchSource:0}: Error finding container 3cb3dceeea80701a7eeda43c4761453bf4e3d7769f998c16ec2f51a4d6a7161c: Status 404 returned error can't find the container with id 3cb3dceeea80701a7eeda43c4761453bf4e3d7769f998c16ec2f51a4d6a7161c Jul 03 11:12:19.238401 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.238381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:19.247013 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.246990 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a62c8c1_f2e5_4a9a_bda4_530482895a9a.slice/crio-9c356dea5e002138b529d360ff0c4053b854261d25326fee9e0777f076b11555 WatchSource:0}: Error finding container 9c356dea5e002138b529d360ff0c4053b854261d25326fee9e0777f076b11555: Status 404 returned error can't find the container with id 9c356dea5e002138b529d360ff0c4053b854261d25326fee9e0777f076b11555 Jul 03 11:12:19.253410 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.253389 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" Jul 03 11:12:19.260875 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.260846 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e5067b3_011f_4a3b_a128_86ce147a1c69.slice/crio-1032fe537ed59c99bcf2f4f9eb5c8caa30694a4059cc957d112277ca6ae7b29e WatchSource:0}: Error finding container 1032fe537ed59c99bcf2f4f9eb5c8caa30694a4059cc957d112277ca6ae7b29e: Status 404 returned error can't find the container with id 1032fe537ed59c99bcf2f4f9eb5c8caa30694a4059cc957d112277ca6ae7b29e Jul 03 11:12:19.266263 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.266245 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8d5tj" Jul 03 11:12:19.273170 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.273146 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a6b54da_ff3a_4ad1_868b_1f06684af304.slice/crio-dbcc2ff245c4760d5a3c762b8b103d1a28c4195c2c398f313fe33bada8a1d3e2 WatchSource:0}: Error finding container dbcc2ff245c4760d5a3c762b8b103d1a28c4195c2c398f313fe33bada8a1d3e2: Status 404 returned error can't find the container with id dbcc2ff245c4760d5a3c762b8b103d1a28c4195c2c398f313fe33bada8a1d3e2 Jul 03 11:12:19.284268 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.284248 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8tn46" Jul 03 11:12:19.285153 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.285131 2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 11:12:19.292070 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.292047 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc69354_77c5_484c_b655_7ff694e5e9d9.slice/crio-f9c7646d7834398f69a41981fb925b08cc1e07abcb353f4a6cff7a9b807e5ec7 WatchSource:0}: Error finding container f9c7646d7834398f69a41981fb925b08cc1e07abcb353f4a6cff7a9b807e5ec7: Status 404 returned error can't find the container with id f9c7646d7834398f69a41981fb925b08cc1e07abcb353f4a6cff7a9b807e5ec7 Jul 03 11:12:19.303925 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.303905 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-q5wgx" Jul 03 11:12:19.308677 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.308658 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:19.311816 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.311793 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c03a68_858d_4402_98e6_6ce9a2b9a6b2.slice/crio-818fd8b29975f4c41c57d10bd797b025784960ce60f6541f6771b51330de6285 WatchSource:0}: Error finding container 818fd8b29975f4c41c57d10bd797b025784960ce60f6541f6771b51330de6285: Status 404 returned error can't find the container with id 818fd8b29975f4c41c57d10bd797b025784960ce60f6541f6771b51330de6285 Jul 03 11:12:19.316476 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:19.316455 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod051916bd_975d_49bf_810c_768ed087aa68.slice/crio-d83e874a4b66ce571067eae586a81cd8b5531bd514d612def7d453beab98d115 WatchSource:0}: Error finding container d83e874a4b66ce571067eae586a81cd8b5531bd514d612def7d453beab98d115: Status 404 returned error can't find the container with id d83e874a4b66ce571067eae586a81cd8b5531bd514d612def7d453beab98d115 Jul 03 11:12:19.568802 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.568703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:19.568974 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.568804 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:19.568974 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.568877 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:19.568974 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.568908 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:19.568974 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.568946 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs podName:e6925168-d48c-405d-8a6c-30384df3214c nodeName:}" failed. No retries permitted until 2026-07-03 11:12:20.568928315 +0000 UTC m=+3.131603825 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs") pod "network-metrics-daemon-dqt7b" (UID: "e6925168-d48c-405d-8a6c-30384df3214c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:19.568974 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.568964 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret podName:0b814191-d67d-47b0-8c3f-af739872704b nodeName:}" failed. No retries permitted until 2026-07-03 11:12:20.568957156 +0000 UTC m=+3.131632647 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret") pod "global-pull-secret-syncer-q2wrq" (UID: "0b814191-d67d-47b0-8c3f-af739872704b") : object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:19.669446 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.669394 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:19.669679 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.669592 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 11:12:19.669679 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.669612 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 11:12:19.669679 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.669620 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2cnmr for pod openshift-network-diagnostics/network-check-target-76w8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:19.669679 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:19.669678 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr podName:69bdf6b0-795e-4c72-b8f6-dd577472c2de nodeName:}" failed. No retries permitted until 2026-07-03 11:12:20.669661967 +0000 UTC m=+3.232337471 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cnmr" (UniqueName: "kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr") pod "network-check-target-76w8c" (UID: "69bdf6b0-795e-4c72-b8f6-dd577472c2de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:19.955029 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.954940 2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 11:12:19.971956 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.971912 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 11:07:18 +0000 UTC" deadline="2028-01-18 01:03:31.698373073 +0000 UTC" Jul 03 11:12:19.971956 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:19.971954 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13525h51m11.726424136s" Jul 03 11:12:20.013272 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.013199 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tn46" event={"ID":"5dc69354-77c5-484c-b655-7ff694e5e9d9","Type":"ContainerStarted","Data":"f9c7646d7834398f69a41981fb925b08cc1e07abcb353f4a6cff7a9b807e5ec7"} Jul 03 11:12:20.022040 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.022002 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mc48b" event={"ID":"b4cfd292-27df-444a-b0af-43067d41ee34","Type":"ContainerStarted","Data":"b659029b73c978b1a23b31d5cc2fa26849340f9fd16e084eb995e2a292bdf5d2"} Jul 03 11:12:20.035590 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.035488 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" event={"ID":"629b92be-f77d-4067-8732-cea6d1e6c2dd","Type":"ContainerStarted","Data":"849527f6d8db9b1e2f12cff804af3b8f9e05519d3af31e7b5243e3e0bc4f8b0f"} Jul 03 11:12:20.046974 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.046934 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" event={"ID":"016eda45f6076f69f860c98ef70bfd7f","Type":"ContainerStarted","Data":"d444752ed1e981a80e14242c0017b42fde820b4f38bd0633a564bfc6384e14f7"} Jul 03 11:12:20.066595 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.066555 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" event={"ID":"36c40a12ef0a62b4aa40697a8c2cbf13","Type":"ContainerStarted","Data":"936a6b5c57ec0a404c2f5b5165fb850c2dfde9a508ad6b955a4169a960425416"} Jul 03 11:12:20.096815 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.096694 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q5wgx" event={"ID":"60c03a68-858d-4402-98e6-6ce9a2b9a6b2","Type":"ContainerStarted","Data":"818fd8b29975f4c41c57d10bd797b025784960ce60f6541f6771b51330de6285"} Jul 03 11:12:20.122350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.122300 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8d5tj" event={"ID":"1a6b54da-ff3a-4ad1-868b-1f06684af304","Type":"ContainerStarted","Data":"dbcc2ff245c4760d5a3c762b8b103d1a28c4195c2c398f313fe33bada8a1d3e2"} Jul 03 11:12:20.161247 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.161207 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" event={"ID":"6e5067b3-011f-4a3b-a128-86ce147a1c69","Type":"ContainerStarted","Data":"1032fe537ed59c99bcf2f4f9eb5c8caa30694a4059cc957d112277ca6ae7b29e"} Jul 03 11:12:20.176674 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.176625 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"9c356dea5e002138b529d360ff0c4053b854261d25326fee9e0777f076b11555"} Jul 03 11:12:20.194516 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.194470 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ft7vq" event={"ID":"6d0b8313-5882-4bcb-8192-91a980250a70","Type":"ContainerStarted","Data":"3cb3dceeea80701a7eeda43c4761453bf4e3d7769f998c16ec2f51a4d6a7161c"} Jul 03 11:12:20.214587 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.214494 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wkzwc" event={"ID":"051916bd-975d-49bf-810c-768ed087aa68","Type":"ContainerStarted","Data":"d83e874a4b66ce571067eae586a81cd8b5531bd514d612def7d453beab98d115"} Jul 03 11:12:20.578605 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.577785 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:20.578605 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.577864 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:20.578605 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.578035 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:20.578605 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.578094 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret podName:0b814191-d67d-47b0-8c3f-af739872704b nodeName:}" failed. No retries permitted until 2026-07-03 11:12:22.578076774 +0000 UTC m=+5.140752271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret") pod "global-pull-secret-syncer-q2wrq" (UID: "0b814191-d67d-47b0-8c3f-af739872704b") : object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:20.578605 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.578514 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:20.578605 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.578568 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs podName:e6925168-d48c-405d-8a6c-30384df3214c nodeName:}" failed. No retries permitted until 2026-07-03 11:12:22.578551508 +0000 UTC m=+5.141227001 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs") pod "network-metrics-daemon-dqt7b" (UID: "e6925168-d48c-405d-8a6c-30384df3214c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:20.634876 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.634841 2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160" Jul 03 11:12:20.679184 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.679148 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:20.679368 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.679345 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 11:12:20.679368 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.679366 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 11:12:20.679528 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.679378 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2cnmr for pod openshift-network-diagnostics/network-check-target-76w8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:20.679528 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.679449 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr podName:69bdf6b0-795e-4c72-b8f6-dd577472c2de nodeName:}" failed. No retries permitted until 2026-07-03 11:12:22.679417457 +0000 UTC m=+5.242092953 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cnmr" (UniqueName: "kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr") pod "network-check-target-76w8c" (UID: "69bdf6b0-795e-4c72-b8f6-dd577472c2de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:20.973046 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.972896 2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-02 11:07:18 +0000 UTC" deadline="2028-02-08 06:08:25.280253195 +0000 UTC" Jul 03 11:12:20.973046 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.972945 2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14034h56m4.307312523s" Jul 03 11:12:20.990047 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.988102 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:20.990047 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.988234 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:20.990047 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.988685 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:20.990047 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.988777 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:20.990047 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:20.988839 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:20.990047 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:20.988918 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:22.596548 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:22.596488 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:22.597039 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:22.596564 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:22.597039 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.596699 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:22.597039 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.596763 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret podName:0b814191-d67d-47b0-8c3f-af739872704b nodeName:}" failed. No retries permitted until 2026-07-03 11:12:26.596744956 +0000 UTC m=+9.159420450 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret") pod "global-pull-secret-syncer-q2wrq" (UID: "0b814191-d67d-47b0-8c3f-af739872704b") : object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:22.597157 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.597101 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:22.597157 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.597138 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs podName:e6925168-d48c-405d-8a6c-30384df3214c nodeName:}" failed. No retries permitted until 2026-07-03 11:12:26.597126574 +0000 UTC m=+9.159802063 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs") pod "network-metrics-daemon-dqt7b" (UID: "e6925168-d48c-405d-8a6c-30384df3214c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:22.697311 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:22.697272 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:22.697483 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.697420 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 11:12:22.697560 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.697486 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 11:12:22.697560 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.697502 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2cnmr for pod openshift-network-diagnostics/network-check-target-76w8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:22.697662 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.697571 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr podName:69bdf6b0-795e-4c72-b8f6-dd577472c2de nodeName:}" failed. No retries permitted until 2026-07-03 11:12:26.697550875 +0000 UTC m=+9.260226369 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cnmr" (UniqueName: "kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr") pod "network-check-target-76w8c" (UID: "69bdf6b0-795e-4c72-b8f6-dd577472c2de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:22.988877 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:22.988563 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:22.988877 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.988727 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:22.989103 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:22.989063 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:22.989323 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.989156 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:22.989323 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:22.989206 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:22.989323 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:22.989289 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:24.988895 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:24.988668 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:24.989374 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:24.988667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:24.989374 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:24.989016 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:24.989374 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:24.988739 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:24.989374 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:24.989100 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:24.989374 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:24.989171 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:26.633470 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:26.633368 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:26.633891 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:26.633536 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:26.633891 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.633676 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:26.633891 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.633753 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret podName:0b814191-d67d-47b0-8c3f-af739872704b nodeName:}" failed. No retries permitted until 2026-07-03 11:12:34.63373135 +0000 UTC m=+17.196406901 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret") pod "global-pull-secret-syncer-q2wrq" (UID: "0b814191-d67d-47b0-8c3f-af739872704b") : object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:26.634346 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.634249 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:26.634346 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.634313 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs podName:e6925168-d48c-405d-8a6c-30384df3214c nodeName:}" failed. No retries permitted until 2026-07-03 11:12:34.634288301 +0000 UTC m=+17.196963792 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs") pod "network-metrics-daemon-dqt7b" (UID: "e6925168-d48c-405d-8a6c-30384df3214c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:26.734488 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:26.734445 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:26.734658 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.734622 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 11:12:26.734658 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.734652 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 11:12:26.734742 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.734668 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2cnmr for pod openshift-network-diagnostics/network-check-target-76w8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:26.734742 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.734728 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr podName:69bdf6b0-795e-4c72-b8f6-dd577472c2de nodeName:}" failed. No retries permitted until 2026-07-03 11:12:34.734708567 +0000 UTC m=+17.297384062 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cnmr" (UniqueName: "kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr") pod "network-check-target-76w8c" (UID: "69bdf6b0-795e-4c72-b8f6-dd577472c2de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:26.989064 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:26.988617 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:26.989064 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:26.988667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:26.989064 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.988722 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:26.989064 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.988835 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:26.989064 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:26.988906 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:26.989064 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:26.989018 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:28.988064 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:28.987986 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:28.988567 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:28.988112 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:28.988567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:28.988172 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:28.988567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:28.988189 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:28.988567 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:28.988245 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:28.988567 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:28.988331 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:30.988475 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:30.988420 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:30.988828 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:30.988420 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:30.988828 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:30.988545 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:30.988828 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:30.988673 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:30.988828 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:30.988424 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:30.988828 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:30.988775 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:32.988645 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:32.988602 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:32.988645 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:32.988649 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:32.989143 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:32.988608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:32.989143 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:32.988755 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:32.989143 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:32.988869 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:32.989143 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:32.988949 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:34.697686 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:34.697643 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:34.698179 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:34.697730 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:34.698179 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.697825 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:34.698179 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.697823 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:34.698179 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.697887 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret podName:0b814191-d67d-47b0-8c3f-af739872704b nodeName:}" failed. No retries permitted until 2026-07-03 11:12:50.697871226 +0000 UTC m=+33.260546729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret") pod "global-pull-secret-syncer-q2wrq" (UID: "0b814191-d67d-47b0-8c3f-af739872704b") : object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:34.698179 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.697906 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs podName:e6925168-d48c-405d-8a6c-30384df3214c nodeName:}" failed. No retries permitted until 2026-07-03 11:12:50.697897234 +0000 UTC m=+33.260572729 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs") pod "network-metrics-daemon-dqt7b" (UID: "e6925168-d48c-405d-8a6c-30384df3214c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:34.798397 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:34.798358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:34.798586 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.798549 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 11:12:34.798586 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.798580 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 11:12:34.798680 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.798595 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2cnmr for pod openshift-network-diagnostics/network-check-target-76w8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:34.798680 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.798654 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr podName:69bdf6b0-795e-4c72-b8f6-dd577472c2de nodeName:}" failed. No retries permitted until 2026-07-03 11:12:50.798637547 +0000 UTC m=+33.361313036 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cnmr" (UniqueName: "kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr") pod "network-check-target-76w8c" (UID: "69bdf6b0-795e-4c72-b8f6-dd577472c2de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:34.988633 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:34.988545 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:34.988633 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:34.988597 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:34.988897 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:34.988651 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:34.988897 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.988669 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:34.988897 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.988746 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:34.988897 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:34.988798 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:36.988661 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:36.988619 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:36.989113 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:36.988666 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:36.989113 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:36.988722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:36.989113 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:36.988739 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:36.989113 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:36.988837 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:36.989113 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:36.988926 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:38.988094 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:38.988059 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:38.988567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:38.988177 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:38.988567 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:38.988188 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:38.988567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:38.988221 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:38.988567 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:38.988308 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:38.988567 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:38.988403 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:40.266779 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.266608 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" event={"ID":"629b92be-f77d-4067-8732-cea6d1e6c2dd","Type":"ContainerStarted","Data":"e13a85924e91c3b56c6b8be7cad6ee2b996cb715975ca6afc8828701ecd8725f"} Jul 03 11:12:40.268707 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.268406 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" event={"ID":"36c40a12ef0a62b4aa40697a8c2cbf13","Type":"ContainerStarted","Data":"b344c7063794c4f0c3d2c16e13c59aebd04a3324877600c933c0bd12c80e5c17"} Jul 03 11:12:40.270863 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.270837 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-q5wgx" event={"ID":"60c03a68-858d-4402-98e6-6ce9a2b9a6b2","Type":"ContainerStarted","Data":"878e060d9a71cf26b2b279b9e5f6690cd65108b28dd1da44c74c9e1d2beb286e"} Jul 03 11:12:40.272672 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.272648 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:12:40.273148 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.273121 2581 generic.go:356] "Generic (PLEG): container finished" podID="5a62c8c1-f2e5-4a9a-bda4-530482895a9a" containerID="fb67af73bcbe6b26feb594d9155ddb47937adbd8ab61f809c291d7ee31482b41" exitCode=1 Jul 03 11:12:40.273248 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.273165 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"be9d7741ff8fdd1d388c1b588319fa69c8461d8a0c28c55667e5a16d0ce99fce"} Jul 03 11:12:40.273248 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.273234 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerDied","Data":"fb67af73bcbe6b26feb594d9155ddb47937adbd8ab61f809c291d7ee31482b41"} Jul 03 11:12:40.273354 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.273286 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"e4fbba0a689660235fa83dacedd744df5bca1cfce47203052e4f7ea644b55a27"} Jul 03 11:12:40.285199 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.285143 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-w4vkh" podStartSLOduration=2.049645826 podStartE2EDuration="22.285101194s" totalImagesPullingTime="20.235455368s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:40.284368694 +0000 UTC m=+22.847044219" watchObservedRunningTime="2026-07-03 11:12:40.285101194 +0000 UTC m=+22.847776706" Jul 03 11:12:40.297570 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.297520 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-89.ec2.internal" podStartSLOduration=22.29750269 podStartE2EDuration="22.29750269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:40.297237074 +0000 UTC m=+22.859912586" watchObservedRunningTime="2026-07-03 11:12:40.29750269 +0000 UTC m=+22.860178201" Jul 03 11:12:40.320870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.320572 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-q5wgx" podStartSLOduration=1.938775978 podStartE2EDuration="22.320549562s" totalImagesPullingTime="20.381773584s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:40.319885329 +0000 UTC m=+22.882560840" watchObservedRunningTime="2026-07-03 11:12:40.320549562 +0000 UTC m=+22.883225077" Jul 03 11:12:40.988477 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.988227 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:40.988477 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.988273 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:40.988477 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:40.988279 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:40.988712 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:40.988528 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:40.988712 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:40.988615 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:40.988712 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:40.988680 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:41.276718 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.276635 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8d5tj" event={"ID":"1a6b54da-ff3a-4ad1-868b-1f06684af304","Type":"ContainerStarted","Data":"d5223289d65a4ff7c679c954b7452513fdc72da4032ff7f249614d176e2b1c5f"} Jul 03 11:12:41.278152 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.278131 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" event={"ID":"6e5067b3-011f-4a3b-a128-86ce147a1c69","Type":"ContainerStarted","Data":"c8e343aa6c0548fc01b304e08c2b3e0a01c3eafaa081746c716f19585f320268"} Jul 03 11:12:41.280965 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.280946 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:12:41.281297 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.281278 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"fff33573f1c46762581ee0c0bfd007f728f4a6728d2e9ff46fa0e6f023de63ac"} Jul 03 11:12:41.281369 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.281341 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"2a221f7543acbba284a50c8a410ef3348371dd6ffe9f4033eaf3a60b1a0a049e"} Jul 03 11:12:41.281427 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.281378 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"079d76e9e899be8943a044789f893b5bf6868aaa1f15a004e48b9d0c3e1824db"} Jul 03 11:12:41.282652 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.282622 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-ft7vq" event={"ID":"6d0b8313-5882-4bcb-8192-91a980250a70","Type":"ContainerStarted","Data":"6d2c7b7395494f3ccb25fe38390a7f05a399092e41dc10a998fc4b5ef8670664"} Jul 03 11:12:41.284059 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.284036 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wkzwc" event={"ID":"051916bd-975d-49bf-810c-768ed087aa68","Type":"ContainerStarted","Data":"12cb19304d9bb893064b0aadea4fd92dd751f51aa5d86ffdd19e87fd31ff7c1b"} Jul 03 11:12:41.285474 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.285453 2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc69354-77c5-484c-b655-7ff694e5e9d9" containerID="2114af2238186871996287238533e52d7e901317135520abb200898c757d6733" exitCode=0 Jul 03 11:12:41.285545 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.285522 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tn46" event={"ID":"5dc69354-77c5-484c-b655-7ff694e5e9d9","Type":"ContainerDied","Data":"2114af2238186871996287238533e52d7e901317135520abb200898c757d6733"} Jul 03 11:12:41.286974 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.286952 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mc48b" event={"ID":"b4cfd292-27df-444a-b0af-43067d41ee34","Type":"ContainerStarted","Data":"77927da0d862211f095910f2cb68b7a70bf3bbf4fcd498abc7edca9dcffff900"} Jul 03 11:12:41.288359 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.288336 2581 generic.go:356] "Generic (PLEG): container finished" podID="016eda45f6076f69f860c98ef70bfd7f" containerID="807ae9f021384208a09a505a7bdb60b1becbb96b89db4494e00f93f0a30f3f0d" exitCode=0 Jul 03 11:12:41.288444 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.288406 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" event={"ID":"016eda45f6076f69f860c98ef70bfd7f","Type":"ContainerDied","Data":"807ae9f021384208a09a505a7bdb60b1becbb96b89db4494e00f93f0a30f3f0d"} Jul 03 11:12:41.293029 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.292994 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8d5tj" podStartSLOduration=3.134615497 podStartE2EDuration="23.292982282s" totalImagesPullingTime="20.158366785s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:41.292730939 +0000 UTC m=+23.855406453" watchObservedRunningTime="2026-07-03 11:12:41.292982282 +0000 UTC m=+23.855657793" Jul 03 11:12:41.319183 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.319132 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-ft7vq" podStartSLOduration=3.120343454 podStartE2EDuration="23.319116385s" totalImagesPullingTime="20.198772931s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:41.318769228 +0000 UTC m=+23.881444739" watchObservedRunningTime="2026-07-03 11:12:41.319116385 +0000 UTC m=+23.881791897" Jul 03 11:12:41.351704 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.351641 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mc48b" podStartSLOduration=3.122597553 podStartE2EDuration="23.35162086s" totalImagesPullingTime="20.229023307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:41.35114735 +0000 UTC m=+23.913822860" watchObservedRunningTime="2026-07-03 11:12:41.35162086 +0000 UTC m=+23.914296373" Jul 03 11:12:41.368964 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.368910 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wkzwc" podStartSLOduration=3.254230637 podStartE2EDuration="23.368887237s" totalImagesPullingTime="20.1146566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:41.368552293 +0000 UTC m=+23.931227804" watchObservedRunningTime="2026-07-03 11:12:41.368887237 +0000 UTC m=+23.931562750" Jul 03 11:12:41.889503 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.889470 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:41.890014 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:41.889994 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:42.043762 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.043732 2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Jul 03 11:12:42.292381 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.292298 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" event={"ID":"6e5067b3-011f-4a3b-a128-86ce147a1c69","Type":"ContainerStarted","Data":"9a6b8ad2761f03085f00f534efb328acbeac9c154d6c2f25c973566746f54bdd"} Jul 03 11:12:42.294003 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.293963 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" event={"ID":"016eda45f6076f69f860c98ef70bfd7f","Type":"ContainerStarted","Data":"724714bb7d78771c69a8963cb9076b039fbf96cea5d3e2415d388f89ee20f19b"} Jul 03 11:12:42.294541 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.294514 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:42.294811 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.294798 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wkzwc" Jul 03 11:12:42.308945 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.308901 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-89.ec2.internal" podStartSLOduration=24.308888991 podStartE2EDuration="24.308888991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:42.308564123 +0000 UTC m=+24.871239635" watchObservedRunningTime="2026-07-03 11:12:42.308888991 +0000 UTC m=+24.871564503" Jul 03 11:12:42.938110 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.938007 2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-03T11:12:42.043755485Z","UUID":"edaca3b1-fcd6-42a9-9a62-4c48db83309b","Handler":null,"Name":"","Endpoint":""} Jul 03 11:12:42.939680 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.939659 2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Jul 03 11:12:42.939820 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.939706 2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Jul 03 11:12:42.988573 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.988497 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:42.988739 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.988507 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:42.988739 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:42.988625 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:42.988739 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:42.988701 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:42.988739 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:42.988509 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:42.988969 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:42.988802 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:43.300312 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:43.300235 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:12:43.300803 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:43.300775 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"99dc9f4ba60a27a6d71c02f0053c33a49685f603f3ac7fabc1396e9a8d1474ca"} Jul 03 11:12:44.304586 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:44.304548 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" event={"ID":"6e5067b3-011f-4a3b-a128-86ce147a1c69","Type":"ContainerStarted","Data":"02a79804134df2ac529f5f9662ad45ba7122eafc654fb61f017766cfb51b8d18"} Jul 03 11:12:44.323536 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:44.323489 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-mmmr5" podStartSLOduration=3.407677261 podStartE2EDuration="26.323473031s" totalImagesPullingTime="22.91579577s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:44.323378745 +0000 UTC m=+26.886054271" watchObservedRunningTime="2026-07-03 11:12:44.323473031 +0000 UTC m=+26.886148542" Jul 03 11:12:44.988451 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:44.988400 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:44.988643 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:44.988625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:44.988718 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:44.988648 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:44.988777 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:44.988753 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:44.988922 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:44.988890 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:44.989004 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:44.988987 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:45.311106 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:45.311076 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:12:45.311773 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:45.311409 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"4eb60ce87cffe1cad60dce218d8920af709b52c724c4957fbb0d2ab23d167320"} Jul 03 11:12:45.311876 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:45.311861 2581 scope.go:117] "RemoveContainer" containerID="fb67af73bcbe6b26feb594d9155ddb47937adbd8ab61f809c291d7ee31482b41" Jul 03 11:12:46.318453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.318259 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:12:46.318961 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.318800 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" event={"ID":"5a62c8c1-f2e5-4a9a-bda4-530482895a9a","Type":"ContainerStarted","Data":"7bfc4d9b63cc9729ed071b2a3ea9b2768e91bdc4c0d29bdc6bb8df4c2c7b9d16"} Jul 03 11:12:46.319278 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.319177 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:46.319278 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.319258 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:46.319427 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.319307 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:46.337040 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.337010 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:46.337409 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.337390 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:12:46.349165 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.349105 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" podStartSLOduration=7.676976751 podStartE2EDuration="28.349088146s" totalImagesPullingTime="20.672111395s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:12:46.347367682 +0000 UTC m=+28.910043229" watchObservedRunningTime="2026-07-03 11:12:46.349088146 +0000 UTC m=+28.911763657" Jul 03 11:12:46.988656 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.988615 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:46.988887 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.988866 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:46.988983 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:46.988878 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:46.988983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:46.988904 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:46.989088 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:46.989031 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:46.989157 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:46.989136 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:47.311178 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:47.311095 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2wrq"] Jul 03 11:12:47.311985 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:47.311956 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqt7b"] Jul 03 11:12:47.312563 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:47.312541 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-76w8c"] Jul 03 11:12:47.321226 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:47.321194 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:47.321694 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:47.321199 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:47.321694 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:47.321305 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:47.321694 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:47.321199 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:47.321839 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:47.321818 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:47.322180 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:47.322140 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:48.325604 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:48.325574 2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc69354-77c5-484c-b655-7ff694e5e9d9" containerID="fb1b5515ec5e444c1f862f6c6258b6a89c9c963a212c43545d46bc04c295c4ce" exitCode=0 Jul 03 11:12:48.326042 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:48.325664 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tn46" event={"ID":"5dc69354-77c5-484c-b655-7ff694e5e9d9","Type":"ContainerDied","Data":"fb1b5515ec5e444c1f862f6c6258b6a89c9c963a212c43545d46bc04c295c4ce"} Jul 03 11:12:48.988351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:48.988312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:48.988499 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:48.988316 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:48.988499 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:48.988312 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:48.988499 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:48.988477 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:48.988614 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:48.988524 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:48.988648 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:48.988615 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:49.330209 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:49.330178 2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc69354-77c5-484c-b655-7ff694e5e9d9" containerID="4b2cfd05efb9b8875af082f871d972ab31b17879130266d8502df84e5219bc46" exitCode=0 Jul 03 11:12:49.330581 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:49.330239 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tn46" event={"ID":"5dc69354-77c5-484c-b655-7ff694e5e9d9","Type":"ContainerDied","Data":"4b2cfd05efb9b8875af082f871d972ab31b17879130266d8502df84e5219bc46"} Jul 03 11:12:50.335121 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:50.334899 2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc69354-77c5-484c-b655-7ff694e5e9d9" containerID="a9cc85f97b9c473e0f93d10996bb15d205d9617cbc6a41f1b5d4062a67947145" exitCode=0 Jul 03 11:12:50.335121 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:50.334964 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tn46" event={"ID":"5dc69354-77c5-484c-b655-7ff694e5e9d9","Type":"ContainerDied","Data":"a9cc85f97b9c473e0f93d10996bb15d205d9617cbc6a41f1b5d4062a67947145"} Jul 03 11:12:50.715400 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:50.715365 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:50.715612 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:50.715420 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:50.715612 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.715536 2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:50.715612 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.715543 2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:50.715612 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.715588 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret podName:0b814191-d67d-47b0-8c3f-af739872704b nodeName:}" failed. No retries permitted until 2026-07-03 11:13:22.715575581 +0000 UTC m=+65.278251071 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret") pod "global-pull-secret-syncer-q2wrq" (UID: "0b814191-d67d-47b0-8c3f-af739872704b") : object "kube-system"/"original-pull-secret" not registered Jul 03 11:12:50.715612 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.715604 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs podName:e6925168-d48c-405d-8a6c-30384df3214c nodeName:}" failed. No retries permitted until 2026-07-03 11:13:22.7155965 +0000 UTC m=+65.278271989 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs") pod "network-metrics-daemon-dqt7b" (UID: "e6925168-d48c-405d-8a6c-30384df3214c") : object "openshift-multus"/"metrics-daemon-secret" not registered Jul 03 11:12:50.816310 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:50.816281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:50.816459 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.816444 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Jul 03 11:12:50.816496 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.816466 2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Jul 03 11:12:50.816496 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.816477 2581 projected.go:196] Error preparing data for projected volume kube-api-access-2cnmr for pod openshift-network-diagnostics/network-check-target-76w8c: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:50.816552 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.816529 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr podName:69bdf6b0-795e-4c72-b8f6-dd577472c2de nodeName:}" failed. No retries permitted until 2026-07-03 11:13:22.81651644 +0000 UTC m=+65.379191950 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2cnmr" (UniqueName: "kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr") pod "network-check-target-76w8c" (UID: "69bdf6b0-795e-4c72-b8f6-dd577472c2de") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Jul 03 11:12:50.988584 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:50.988497 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:50.988584 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:50.988538 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:50.988584 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:50.988560 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:50.988771 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.988622 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-q2wrq" podUID="0b814191-d67d-47b0-8c3f-af739872704b" Jul 03 11:12:50.988771 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.988739 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-dqt7b" podUID="e6925168-d48c-405d-8a6c-30384df3214c" Jul 03 11:12:50.988832 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:50.988811 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-76w8c" podUID="69bdf6b0-795e-4c72-b8f6-dd577472c2de" Jul 03 11:12:52.691448 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.691412 2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-89.ec2.internal" event="NodeReady" Jul 03 11:12:52.692022 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.691573 2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready" Jul 03 11:12:52.730039 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.730009 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-846768dbb5-2blkk"] Jul 03 11:12:52.733745 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.733723 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.736172 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.736148 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" Jul 03 11:12:52.736287 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.736216 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58shg\"" Jul 03 11:12:52.736605 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.736416 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" Jul 03 11:12:52.736605 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.736497 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" Jul 03 11:12:52.741744 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.741724 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg"] Jul 03 11:12:52.744259 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.744054 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" Jul 03 11:12:52.745909 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.745890 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv"] Jul 03 11:12:52.746015 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.745998 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg" Jul 03 11:12:52.748544 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.748512 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.749222 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.749061 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.749322 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.749287 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-7wpk5"] Jul 03 11:12:52.749530 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.749504 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-62tdj\"" Jul 03 11:12:52.749628 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.749577 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:52.752532 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.752492 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-678cfbb59c-mm6jx"] Jul 03 11:12:52.752697 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.752676 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.753216 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.752842 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\"" Jul 03 11:12:52.753216 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.753119 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.753373 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.753323 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.753893 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.753708 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-czfxb\"" Jul 03 11:12:52.755841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.755420 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.756222 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.756203 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\"" Jul 03 11:12:52.756330 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.756307 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-wgbsx\"" Jul 03 11:12:52.756464 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.756420 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm"] Jul 03 11:12:52.757422 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.757039 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.757422 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.757066 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.758265 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.758246 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\"" Jul 03 11:12:52.761997 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.761104 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh"] Jul 03 11:12:52.761997 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.761992 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\"" Jul 03 11:12:52.762546 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.762524 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.766135 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.763916 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lxj4p\"" Jul 03 11:12:52.766135 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.765556 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\"" Jul 03 11:12:52.766135 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.765671 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.769966 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.769940 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\"" Jul 03 11:12:52.771224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.770343 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.771224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.770869 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\"" Jul 03 11:12:52.771224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.770907 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.771224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.770874 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\"" Jul 03 11:12:52.774003 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.773983 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-x6fpk\"" Jul 03 11:12:52.777455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.774310 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\"" Jul 03 11:12:52.777455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.775669 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846768dbb5-2blkk"] Jul 03 11:12:52.777455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.775715 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg"] Jul 03 11:12:52.777455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.775738 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv"] Jul 03 11:12:52.777455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.775761 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.777455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.775782 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-p2bsb"] Jul 03 11:12:52.777455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.776286 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:52.777455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.777182 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\"" Jul 03 11:12:52.778952 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.778928 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.779056 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.779020 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:52.779138 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.778953 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm"] Jul 03 11:12:52.779192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.779149 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh"] Jul 03 11:12:52.779192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.779168 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8"] Jul 03 11:12:52.780339 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.780319 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-84wwp\"" Jul 03 11:12:52.780589 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.780571 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\"" Jul 03 11:12:52.780887 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.780869 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.781142 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.781125 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\"" Jul 03 11:12:52.783721 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.783701 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj"] Jul 03 11:12:52.784156 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.784133 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:52.785030 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.785006 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\"" Jul 03 11:12:52.785030 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.785026 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\"" Jul 03 11:12:52.785171 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.785157 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.785227 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.785176 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.785475 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.785448 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-2gjgc\"" Jul 03 11:12:52.788633 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.788185 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.788633 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.788387 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\"" Jul 03 11:12:52.788894 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.788718 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.789058 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.789040 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9"] Jul 03 11:12:52.789289 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.789272 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj" Jul 03 11:12:52.789995 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.789978 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\"" Jul 03 11:12:52.790262 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.790240 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qvzh2\"" Jul 03 11:12:52.793591 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.793567 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-7wpk5"] Jul 03 11:12:52.793691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.793609 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-p2bsb"] Jul 03 11:12:52.793691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.793632 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj"] Jul 03 11:12:52.793691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.793652 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-678cfbb59c-mm6jx"] Jul 03 11:12:52.793691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.793671 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8"] Jul 03 11:12:52.793691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.793694 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9"] Jul 03 11:12:52.793934 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.793774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:52.794361 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.794337 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.794515 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.794500 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.794793 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.794769 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\"" Jul 03 11:12:52.794879 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.794814 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-sjbpq\"" Jul 03 11:12:52.796828 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.796804 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\"" Jul 03 11:12:52.796948 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.796931 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9wzc8\"" Jul 03 11:12:52.797036 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.797020 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\"" Jul 03 11:12:52.830785 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.830764 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-rzn28"] Jul 03 11:12:52.831309 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831288 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-image-registry-private-configuration\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.831405 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831324 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-installation-pull-secrets\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.831405 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831349 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-tmp\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.831405 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831374 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-snapshots\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.831405 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831396 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhdkd\" (UniqueName: \"kubernetes.io/projected/9885b117-aabd-414f-be44-5ee5f7c908eb-kube-api-access-jhdkd\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.831661 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831455 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9885b117-aabd-414f-be44-5ee5f7c908eb-config\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.831661 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831482 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7hqc\" (UniqueName: \"kubernetes.io/projected/e7554f43-afeb-4d23-9530-589db05eeb2f-kube-api-access-p7hqc\") pod \"volume-data-source-validator-5d456fdbcd-2j6rg\" (UID: \"e7554f43-afeb-4d23-9530-589db05eeb2f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg" Jul 03 11:12:52.831661 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831507 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:52.831661 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831561 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxwbn\" (UniqueName: \"kubernetes.io/projected/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-kube-api-access-qxwbn\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.831835 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831698 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.831835 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831784 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.831835 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831812 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-serving-cert\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.831939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.831939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831913 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-ca-trust-extracted\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.832014 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-trusted-ca\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.832014 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.831980 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgvr\" (UniqueName: \"kubernetes.io/projected/ed2f81ba-9422-4403-b06e-fcda340b8afa-kube-api-access-9bgvr\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:52.832014 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832000 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9885b117-aabd-414f-be44-5ee5f7c908eb-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.832151 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832015 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-default-certificate\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.832151 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-certificates\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.832151 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-bound-sa-token\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.832151 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832081 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lczsw\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-kube-api-access-lczsw\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.832151 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832118 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.832391 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832176 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-stats-auth\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.832391 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832207 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.832391 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.832264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w72nb\" (UniqueName: \"kubernetes.io/projected/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-kube-api-access-w72nb\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.834455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.834418 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bfkkq"] Jul 03 11:12:52.834582 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.834543 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rzn28" Jul 03 11:12:52.837673 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.837653 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Jul 03 11:12:52.837812 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.837653 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-w5xwd\"" Jul 03 11:12:52.838013 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.837797 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Jul 03 11:12:52.838141 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.837893 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Jul 03 11:12:52.839451 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.839419 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:52.841733 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.841712 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rzn28"] Jul 03 11:12:52.842326 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.842308 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Jul 03 11:12:52.842663 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.842645 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kggc2\"" Jul 03 11:12:52.842756 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.842734 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Jul 03 11:12:52.849526 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.849496 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bfkkq"] Jul 03 11:12:52.932790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.932756 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmsr9\" (UniqueName: \"kubernetes.io/projected/ae916522-5d55-4fbf-b304-a7f05fae5691-kube-api-access-jmsr9\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:52.932790 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.932793 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c73b128-ad8e-46c4-86bf-1596700247a4-config\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:52.933082 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.932814 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9dd690e-80aa-4886-963a-5616b8041655-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:52.933082 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.932875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.933082 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.932917 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae916522-5d55-4fbf-b304-a7f05fae5691-config-volume\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:52.933082 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.932950 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-trusted-ca\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.933082 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933002 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9885b117-aabd-414f-be44-5ee5f7c908eb-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.933543 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933508 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-default-certificate\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.933668 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933560 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:52.933668 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933603 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:52.933668 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933627 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c70232-5339-4786-bf98-86283e335ea1-cert\") pod \"ingress-canary-rzn28\" (UID: \"c5c70232-5339-4786-bf98-86283e335ea1\") " pod="openshift-ingress-canary/ingress-canary-rzn28" Jul 03 11:12:52.933668 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933651 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjfbd\" (UniqueName: \"kubernetes.io/projected/c1477b24-cb02-4165-925e-8146aa836065-kube-api-access-wjfbd\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:52.933830 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933675 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-certificates\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.933830 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933697 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lczsw\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-kube-api-access-lczsw\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.933830 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933721 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.933830 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933743 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.933830 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933763 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1477b24-cb02-4165-925e-8146aa836065-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:52.933830 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933785 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtmw\" (UniqueName: \"kubernetes.io/projected/c5c70232-5339-4786-bf98-86283e335ea1-kube-api-access-cxtmw\") pod \"ingress-canary-rzn28\" (UID: \"c5c70232-5339-4786-bf98-86283e335ea1\") " pod="openshift-ingress-canary/ingress-canary-rzn28" Jul 03 11:12:52.933830 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c73b128-ad8e-46c4-86bf-1596700247a4-trusted-ca\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:52.934143 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933841 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-image-registry-private-configuration\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.934143 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933862 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhdkd\" (UniqueName: \"kubernetes.io/projected/9885b117-aabd-414f-be44-5ee5f7c908eb-kube-api-access-jhdkd\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.934143 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933861 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-trusted-ca\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.934143 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933883 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-snapshots\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.934143 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.933866 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-trusted-ca-bundle\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.934143 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:52.934032 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 11:12:52.934143 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:52.934041 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:53.434022561 +0000 UTC m=+35.996698068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : configmap references non-existent config key: service-ca.crt Jul 03 11:12:52.934143 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:52.934106 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:53.434088335 +0000 UTC m=+35.996763826 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : secret "router-metrics-certs-default" not found Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934199 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934237 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxwbn\" (UniqueName: \"kubernetes.io/projected/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-kube-api-access-qxwbn\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934278 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934327 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-serving-cert\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934340 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-snapshots\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934358 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934384 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-ca-trust-extracted\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934412 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgvr\" (UniqueName: \"kubernetes.io/projected/ed2f81ba-9422-4403-b06e-fcda340b8afa-kube-api-access-9bgvr\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-bound-sa-token\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934505 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-stats-auth\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934533 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjvqd\" (UniqueName: \"kubernetes.io/projected/1c73b128-ad8e-46c4-86bf-1596700247a4-kube-api-access-gjvqd\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:52.934574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934574 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w72nb\" (UniqueName: \"kubernetes.io/projected/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-kube-api-access-w72nb\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.935105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934601 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c73b128-ad8e-46c4-86bf-1596700247a4-serving-cert\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:52.935105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934623 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-certificates\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.935105 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:52.934703 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 11:12:52.935105 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:52.934716 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846768dbb5-2blkk: secret "image-registry-tls" not found Jul 03 11:12:52.935105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.934981 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-ca-trust-extracted\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.935105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.935088 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-service-ca-bundle\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.935685 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:52.935242 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls podName:db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:53.435226449 +0000 UTC m=+35.997901965 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls") pod "image-registry-846768dbb5-2blkk" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11") : secret "image-registry-tls" not found Jul 03 11:12:52.935685 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.935650 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9885b117-aabd-414f-be44-5ee5f7c908eb-config\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.935802 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.935701 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/950cea78-bd24-478c-810c-e398a8075c7d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:52.935938 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.935884 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzt76\" (UniqueName: \"kubernetes.io/projected/950cea78-bd24-478c-810c-e398a8075c7d-kube-api-access-gzt76\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:52.936011 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.935936 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-installation-pull-secrets\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.936550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.936115 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-tmp\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.936550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.936290 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9885b117-aabd-414f-be44-5ee5f7c908eb-config\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.936550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.936304 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae916522-5d55-4fbf-b304-a7f05fae5691-tmp-dir\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:52.936550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.936348 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7hqc\" (UniqueName: \"kubernetes.io/projected/e7554f43-afeb-4d23-9530-589db05eeb2f-kube-api-access-p7hqc\") pod \"volume-data-source-validator-5d456fdbcd-2j6rg\" (UID: \"e7554f43-afeb-4d23-9530-589db05eeb2f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg" Jul 03 11:12:52.936550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.936375 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:52.936550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.936400 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqdmr\" (UniqueName: \"kubernetes.io/projected/06f56a11-04a4-4b4d-8cfe-7842af7f0bfe-kube-api-access-bqdmr\") pod \"network-check-source-989f8b8dc-wv8nj\" (UID: \"06f56a11-04a4-4b4d-8cfe-7842af7f0bfe\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj" Jul 03 11:12:52.936550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.936451 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1477b24-cb02-4165-925e-8146aa836065-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:52.936550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.936483 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-tmp\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.937059 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:52.936601 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 11:12:52.937059 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:52.936665 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls podName:ed2f81ba-9422-4403-b06e-fcda340b8afa nodeName:}" failed. No retries permitted until 2026-07-03 11:12:53.436650289 +0000 UTC m=+35.999325782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hz5wv" (UID: "ed2f81ba-9422-4403-b06e-fcda340b8afa") : secret "samples-operator-tls" not found Jul 03 11:12:52.938145 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.938120 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9885b117-aabd-414f-be44-5ee5f7c908eb-serving-cert\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.938252 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.938189 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-serving-cert\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.938252 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.938211 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-image-registry-private-configuration\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.938353 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.938322 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-stats-auth\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.938634 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.938612 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-default-certificate\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.939624 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.939603 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-installation-pull-secrets\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.948411 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.948329 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lczsw\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-kube-api-access-lczsw\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.948695 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.948672 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-bound-sa-token\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:52.949008 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.948962 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhdkd\" (UniqueName: \"kubernetes.io/projected/9885b117-aabd-414f-be44-5ee5f7c908eb-kube-api-access-jhdkd\") pod \"service-ca-operator-5d46d9ffd4-vhznm\" (UID: \"9885b117-aabd-414f-be44-5ee5f7c908eb\") " pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:52.949755 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.949715 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w72nb\" (UniqueName: \"kubernetes.io/projected/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-kube-api-access-w72nb\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:52.951004 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.950961 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgvr\" (UniqueName: \"kubernetes.io/projected/ed2f81ba-9422-4403-b06e-fcda340b8afa-kube-api-access-9bgvr\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:52.951460 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.951420 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxwbn\" (UniqueName: \"kubernetes.io/projected/4b09b25f-d483-4e17-8c66-76fc0dbc1bf7-kube-api-access-qxwbn\") pod \"insights-operator-5c847dcfc4-7wpk5\" (UID: \"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7\") " pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:52.952160 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.952143 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7hqc\" (UniqueName: \"kubernetes.io/projected/e7554f43-afeb-4d23-9530-589db05eeb2f-kube-api-access-p7hqc\") pod \"volume-data-source-validator-5d456fdbcd-2j6rg\" (UID: \"e7554f43-afeb-4d23-9530-589db05eeb2f\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg" Jul 03 11:12:52.988113 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.988081 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:12:52.988113 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.988108 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:12:52.988297 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.988193 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:12:52.990691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.990668 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 03 11:12:52.990855 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.990807 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wfk2t\"" Jul 03 11:12:52.991065 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.990878 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 03 11:12:52.991065 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:52.990883 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qx6zp\"" Jul 03 11:12:53.037633 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037597 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjvqd\" (UniqueName: \"kubernetes.io/projected/1c73b128-ad8e-46c4-86bf-1596700247a4-kube-api-access-gjvqd\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.037814 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037641 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c73b128-ad8e-46c4-86bf-1596700247a4-serving-cert\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.037814 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037669 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/950cea78-bd24-478c-810c-e398a8075c7d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:53.037814 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037692 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzt76\" (UniqueName: \"kubernetes.io/projected/950cea78-bd24-478c-810c-e398a8075c7d-kube-api-access-gzt76\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:53.037814 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037720 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae916522-5d55-4fbf-b304-a7f05fae5691-tmp-dir\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:53.037814 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037769 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqdmr\" (UniqueName: \"kubernetes.io/projected/06f56a11-04a4-4b4d-8cfe-7842af7f0bfe-kube-api-access-bqdmr\") pod \"network-check-source-989f8b8dc-wv8nj\" (UID: \"06f56a11-04a4-4b4d-8cfe-7842af7f0bfe\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj" Jul 03 11:12:53.037814 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037793 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1477b24-cb02-4165-925e-8146aa836065-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:53.038105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037819 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmsr9\" (UniqueName: \"kubernetes.io/projected/ae916522-5d55-4fbf-b304-a7f05fae5691-kube-api-access-jmsr9\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:53.038105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037840 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c73b128-ad8e-46c4-86bf-1596700247a4-config\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.038105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037865 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9dd690e-80aa-4886-963a-5616b8041655-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:53.038105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037893 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae916522-5d55-4fbf-b304-a7f05fae5691-config-volume\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:53.038105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037935 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:53.038105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.037961 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:53.038105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038023 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c70232-5339-4786-bf98-86283e335ea1-cert\") pod \"ingress-canary-rzn28\" (UID: \"c5c70232-5339-4786-bf98-86283e335ea1\") " pod="openshift-ingress-canary/ingress-canary-rzn28" Jul 03 11:12:53.038105 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038062 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjfbd\" (UniqueName: \"kubernetes.io/projected/c1477b24-cb02-4165-925e-8146aa836065-kube-api-access-wjfbd\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:53.038507 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038118 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1477b24-cb02-4165-925e-8146aa836065-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:53.038507 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtmw\" (UniqueName: \"kubernetes.io/projected/c5c70232-5339-4786-bf98-86283e335ea1-kube-api-access-cxtmw\") pod \"ingress-canary-rzn28\" (UID: \"c5c70232-5339-4786-bf98-86283e335ea1\") " pod="openshift-ingress-canary/ingress-canary-rzn28" Jul 03 11:12:53.038507 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038193 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c73b128-ad8e-46c4-86bf-1596700247a4-trusted-ca\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.038507 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038239 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:53.038507 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.038351 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 03 11:12:53.038507 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.038393 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert podName:c9dd690e-80aa-4886-963a-5616b8041655 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:53.538381479 +0000 UTC m=+36.101056971 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rcsx9" (UID: "c9dd690e-80aa-4886-963a-5616b8041655") : secret "networking-console-plugin-cert" not found Jul 03 11:12:53.038507 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.038411 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 03 11:12:53.038507 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.038484 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls podName:950cea78-bd24-478c-810c-e398a8075c7d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:53.53846816 +0000 UTC m=+36.101143654 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-28qt8" (UID: "950cea78-bd24-478c-810c-e398a8075c7d") : secret "cluster-monitoring-operator-tls" not found Jul 03 11:12:53.039033 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038616 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ae916522-5d55-4fbf-b304-a7f05fae5691-config-volume\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:53.039033 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.038714 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 11:12:53.039033 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.038766 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls podName:ae916522-5d55-4fbf-b304-a7f05fae5691 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:53.53875114 +0000 UTC m=+36.101426630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls") pod "dns-default-bfkkq" (UID: "ae916522-5d55-4fbf-b304-a7f05fae5691") : secret "dns-default-metrics-tls" not found Jul 03 11:12:53.039033 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038716 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9dd690e-80aa-4886-963a-5616b8041655-nginx-conf\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:53.039033 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038779 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/950cea78-bd24-478c-810c-e398a8075c7d-telemetry-config\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:53.039033 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.038846 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ae916522-5d55-4fbf-b304-a7f05fae5691-tmp-dir\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:53.039276 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.039182 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/c1477b24-cb02-4165-925e-8146aa836065-config\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:53.039657 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.039625 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/1c73b128-ad8e-46c4-86bf-1596700247a4-config\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.039918 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.039897 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/1c73b128-ad8e-46c4-86bf-1596700247a4-trusted-ca\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.041100 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.041076 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/c1477b24-cb02-4165-925e-8146aa836065-serving-cert\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:53.041547 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.041525 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/c5c70232-5339-4786-bf98-86283e335ea1-cert\") pod \"ingress-canary-rzn28\" (UID: \"c5c70232-5339-4786-bf98-86283e335ea1\") " pod="openshift-ingress-canary/ingress-canary-rzn28" Jul 03 11:12:53.041672 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.041654 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/1c73b128-ad8e-46c4-86bf-1596700247a4-serving-cert\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.056534 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.056468 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjvqd\" (UniqueName: \"kubernetes.io/projected/1c73b128-ad8e-46c4-86bf-1596700247a4-kube-api-access-gjvqd\") pod \"console-operator-68b757fb67-p2bsb\" (UID: \"1c73b128-ad8e-46c4-86bf-1596700247a4\") " pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.058986 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.058959 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg" Jul 03 11:12:53.059478 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.059456 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzt76\" (UniqueName: \"kubernetes.io/projected/950cea78-bd24-478c-810c-e398a8075c7d-kube-api-access-gzt76\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:53.059567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.059506 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqdmr\" (UniqueName: \"kubernetes.io/projected/06f56a11-04a4-4b4d-8cfe-7842af7f0bfe-kube-api-access-bqdmr\") pod \"network-check-source-989f8b8dc-wv8nj\" (UID: \"06f56a11-04a4-4b4d-8cfe-7842af7f0bfe\") " pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj" Jul 03 11:12:53.059954 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.059932 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmsr9\" (UniqueName: \"kubernetes.io/projected/ae916522-5d55-4fbf-b304-a7f05fae5691-kube-api-access-jmsr9\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:53.062690 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.062654 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtmw\" (UniqueName: \"kubernetes.io/projected/c5c70232-5339-4786-bf98-86283e335ea1-kube-api-access-cxtmw\") pod \"ingress-canary-rzn28\" (UID: \"c5c70232-5339-4786-bf98-86283e335ea1\") " pod="openshift-ingress-canary/ingress-canary-rzn28" Jul 03 11:12:53.063938 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.063913 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjfbd\" (UniqueName: \"kubernetes.io/projected/c1477b24-cb02-4165-925e-8146aa836065-kube-api-access-wjfbd\") pod \"kube-storage-version-migrator-operator-846bf668f7-ln9mh\" (UID: \"c1477b24-cb02-4165-925e-8146aa836065\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:53.088210 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.088181 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" Jul 03 11:12:53.103649 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.103625 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" Jul 03 11:12:53.109821 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.109799 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" Jul 03 11:12:53.116311 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.116291 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:12:53.128199 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.127976 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj" Jul 03 11:12:53.144674 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.144521 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-rzn28" Jul 03 11:12:53.290235 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.290122 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg"] Jul 03 11:12:53.329697 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.328307 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-5c847dcfc4-7wpk5"] Jul 03 11:12:53.337136 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:53.336122 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b09b25f_d483_4e17_8c66_76fc0dbc1bf7.slice/crio-1f32d06b2df310f279b6d0496eaa85d489fb2990ae7f3a76e00a4dbfe122a709 WatchSource:0}: Error finding container 1f32d06b2df310f279b6d0496eaa85d489fb2990ae7f3a76e00a4dbfe122a709: Status 404 returned error can't find the container with id 1f32d06b2df310f279b6d0496eaa85d489fb2990ae7f3a76e00a4dbfe122a709 Jul 03 11:12:53.347860 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.347821 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg" event={"ID":"e7554f43-afeb-4d23-9530-589db05eeb2f","Type":"ContainerStarted","Data":"ebc867f5b7e6bfed4e7bbe4030b4c14ee20a85ebcca3934c8435d41767a958ee"} Jul 03 11:12:53.349173 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.349116 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" event={"ID":"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7","Type":"ContainerStarted","Data":"1f32d06b2df310f279b6d0496eaa85d489fb2990ae7f3a76e00a4dbfe122a709"} Jul 03 11:12:53.381761 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.381719 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-68b757fb67-p2bsb"] Jul 03 11:12:53.383689 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.383664 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh"] Jul 03 11:12:53.386420 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:53.386391 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c73b128_ad8e_46c4_86bf_1596700247a4.slice/crio-2d94d462c7e5e40a114d4fe5855069b7f38a19541e3496811bbff180f2b48eff WatchSource:0}: Error finding container 2d94d462c7e5e40a114d4fe5855069b7f38a19541e3496811bbff180f2b48eff: Status 404 returned error can't find the container with id 2d94d462c7e5e40a114d4fe5855069b7f38a19541e3496811bbff180f2b48eff Jul 03 11:12:53.386961 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:53.386940 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1477b24_cb02_4165_925e_8146aa836065.slice/crio-927e593c70955a59746fa722551652e26aa9aa2bf85c0cf09e7b025e0d2cabb4 WatchSource:0}: Error finding container 927e593c70955a59746fa722551652e26aa9aa2bf85c0cf09e7b025e0d2cabb4: Status 404 returned error can't find the container with id 927e593c70955a59746fa722551652e26aa9aa2bf85c0cf09e7b025e0d2cabb4 Jul 03 11:12:53.416868 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.416707 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-rzn28"] Jul 03 11:12:53.419716 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.419691 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj"] Jul 03 11:12:53.419936 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:53.419913 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5c70232_5339_4786_bf98_86283e335ea1.slice/crio-d50aa27cf01546899a6690fc16577eada5c48d540ad6568b50bcd2cf7605d973 WatchSource:0}: Error finding container d50aa27cf01546899a6690fc16577eada5c48d540ad6568b50bcd2cf7605d973: Status 404 returned error can't find the container with id d50aa27cf01546899a6690fc16577eada5c48d540ad6568b50bcd2cf7605d973 Jul 03 11:12:53.424903 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:53.424880 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06f56a11_04a4_4b4d_8cfe_7842af7f0bfe.slice/crio-ab5bcfa60d88edf51ae33cf69e24dc845b87a3b209f9c4d71917f8f8eda5bac7 WatchSource:0}: Error finding container ab5bcfa60d88edf51ae33cf69e24dc845b87a3b209f9c4d71917f8f8eda5bac7: Status 404 returned error can't find the container with id ab5bcfa60d88edf51ae33cf69e24dc845b87a3b209f9c4d71917f8f8eda5bac7 Jul 03 11:12:53.442226 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.442199 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:53.442338 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.442228 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:53.442401 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.442351 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 11:12:53.442401 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.442373 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:54.442349874 +0000 UTC m=+37.005025369 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : configmap references non-existent config key: service-ca.crt Jul 03 11:12:53.442531 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.442402 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:54.442385775 +0000 UTC m=+37.005061272 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : secret "router-metrics-certs-default" not found Jul 03 11:12:53.442531 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.442484 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:53.442629 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.442532 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:53.442629 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.442599 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 11:12:53.442629 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.442616 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846768dbb5-2blkk: secret "image-registry-tls" not found Jul 03 11:12:53.442746 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.442636 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 11:12:53.442746 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.442656 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls podName:db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:54.442644768 +0000 UTC m=+37.005320278 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls") pod "image-registry-846768dbb5-2blkk" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11") : secret "image-registry-tls" not found Jul 03 11:12:53.442746 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.442673 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls podName:ed2f81ba-9422-4403-b06e-fcda340b8afa nodeName:}" failed. No retries permitted until 2026-07-03 11:12:54.442662062 +0000 UTC m=+37.005337560 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hz5wv" (UID: "ed2f81ba-9422-4403-b06e-fcda340b8afa") : secret "samples-operator-tls" not found Jul 03 11:12:53.535775 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.535697 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm"] Jul 03 11:12:53.538695 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:12:53.538670 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9885b117_aabd_414f_be44_5ee5f7c908eb.slice/crio-5410cf5c75f590a3c192207b79e174b6d998b5ecdade8515cc648217b6f258b4 WatchSource:0}: Error finding container 5410cf5c75f590a3c192207b79e174b6d998b5ecdade8515cc648217b6f258b4: Status 404 returned error can't find the container with id 5410cf5c75f590a3c192207b79e174b6d998b5ecdade8515cc648217b6f258b4 Jul 03 11:12:53.543352 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.543332 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:53.543456 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.543426 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:53.543494 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:53.543472 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:53.543524 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.543494 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 03 11:12:53.543524 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.543515 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 11:12:53.543587 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.543543 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert podName:c9dd690e-80aa-4886-963a-5616b8041655 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:54.543529006 +0000 UTC m=+37.106204495 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rcsx9" (UID: "c9dd690e-80aa-4886-963a-5616b8041655") : secret "networking-console-plugin-cert" not found Jul 03 11:12:53.543587 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.543560 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls podName:ae916522-5d55-4fbf-b304-a7f05fae5691 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:54.543548813 +0000 UTC m=+37.106224304 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls") pod "dns-default-bfkkq" (UID: "ae916522-5d55-4fbf-b304-a7f05fae5691") : secret "dns-default-metrics-tls" not found Jul 03 11:12:53.543587 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.543561 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 03 11:12:53.543681 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:53.543596 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls podName:950cea78-bd24-478c-810c-e398a8075c7d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:54.54358692 +0000 UTC m=+37.106262410 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-28qt8" (UID: "950cea78-bd24-478c-810c-e398a8075c7d") : secret "cluster-monitoring-operator-tls" not found Jul 03 11:12:54.355252 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.355165 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" event={"ID":"1c73b128-ad8e-46c4-86bf-1596700247a4","Type":"ContainerStarted","Data":"2d94d462c7e5e40a114d4fe5855069b7f38a19541e3496811bbff180f2b48eff"} Jul 03 11:12:54.360933 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.360891 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" event={"ID":"c1477b24-cb02-4165-925e-8146aa836065","Type":"ContainerStarted","Data":"927e593c70955a59746fa722551652e26aa9aa2bf85c0cf09e7b025e0d2cabb4"} Jul 03 11:12:54.367263 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.367042 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" event={"ID":"9885b117-aabd-414f-be44-5ee5f7c908eb","Type":"ContainerStarted","Data":"5410cf5c75f590a3c192207b79e174b6d998b5ecdade8515cc648217b6f258b4"} Jul 03 11:12:54.369785 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.369709 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rzn28" event={"ID":"c5c70232-5339-4786-bf98-86283e335ea1","Type":"ContainerStarted","Data":"d50aa27cf01546899a6690fc16577eada5c48d540ad6568b50bcd2cf7605d973"} Jul 03 11:12:54.374404 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.374377 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj" event={"ID":"06f56a11-04a4-4b4d-8cfe-7842af7f0bfe","Type":"ContainerStarted","Data":"ab5bcfa60d88edf51ae33cf69e24dc845b87a3b209f9c4d71917f8f8eda5bac7"} Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.453700 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.453787 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.453872 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.453901 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.454107 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.454179 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:56.454160551 +0000 UTC m=+39.016836055 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : secret "router-metrics-certs-default" not found Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.454625 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.454644 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846768dbb5-2blkk: secret "image-registry-tls" not found Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.454692 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls podName:db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:56.454674702 +0000 UTC m=+39.017350192 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls") pod "image-registry-846768dbb5-2blkk" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11") : secret "image-registry-tls" not found Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.454750 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.454785 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls podName:ed2f81ba-9422-4403-b06e-fcda340b8afa nodeName:}" failed. No retries permitted until 2026-07-03 11:12:56.454773871 +0000 UTC m=+39.017449365 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hz5wv" (UID: "ed2f81ba-9422-4403-b06e-fcda340b8afa") : secret "samples-operator-tls" not found Jul 03 11:12:54.454906 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.454857 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:56.454836617 +0000 UTC m=+39.017512109 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : configmap references non-existent config key: service-ca.crt Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.555246 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.555286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:54.555349 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.555526 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.555576 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert podName:c9dd690e-80aa-4886-963a-5616b8041655 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:56.55556281 +0000 UTC m=+39.118238299 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rcsx9" (UID: "c9dd690e-80aa-4886-963a-5616b8041655") : secret "networking-console-plugin-cert" not found Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.555618 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.555639 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls podName:ae916522-5d55-4fbf-b304-a7f05fae5691 nodeName:}" failed. No retries permitted until 2026-07-03 11:12:56.555632328 +0000 UTC m=+39.118307817 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls") pod "dns-default-bfkkq" (UID: "ae916522-5d55-4fbf-b304-a7f05fae5691") : secret "dns-default-metrics-tls" not found Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.555670 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 03 11:12:54.555973 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:54.555689 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls podName:950cea78-bd24-478c-810c-e398a8075c7d nodeName:}" failed. No retries permitted until 2026-07-03 11:12:56.555682882 +0000 UTC m=+39.118358371 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-28qt8" (UID: "950cea78-bd24-478c-810c-e398a8075c7d") : secret "cluster-monitoring-operator-tls" not found Jul 03 11:12:56.477688 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:56.477650 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:56.477703 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:56.477780 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.477843 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:00.477820153 +0000 UTC m=+43.040495649 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : configmap references non-existent config key: service-ca.crt Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.477879 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.477887 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.477900 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846768dbb5-2blkk: secret "image-registry-tls" not found Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:56.477908 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.477922 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:00.477908866 +0000 UTC m=+43.040584358 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : secret "router-metrics-certs-default" not found Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.477941 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls podName:db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:00.477931671 +0000 UTC m=+43.040607166 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls") pod "image-registry-846768dbb5-2blkk" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11") : secret "image-registry-tls" not found Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.477974 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 11:12:56.478189 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.478003 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls podName:ed2f81ba-9422-4403-b06e-fcda340b8afa nodeName:}" failed. No retries permitted until 2026-07-03 11:13:00.477995684 +0000 UTC m=+43.040671178 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hz5wv" (UID: "ed2f81ba-9422-4403-b06e-fcda340b8afa") : secret "samples-operator-tls" not found Jul 03 11:12:56.578893 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:56.578848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:12:56.579065 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:56.579012 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:12:56.579065 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.579033 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 03 11:12:56.579065 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:12:56.579045 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:12:56.579214 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.579114 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert podName:c9dd690e-80aa-4886-963a-5616b8041655 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:00.579090782 +0000 UTC m=+43.141766306 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rcsx9" (UID: "c9dd690e-80aa-4886-963a-5616b8041655") : secret "networking-console-plugin-cert" not found Jul 03 11:12:56.579214 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.579140 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 03 11:12:56.579214 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.579172 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 11:12:56.579214 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.579186 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls podName:950cea78-bd24-478c-810c-e398a8075c7d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:00.579171219 +0000 UTC m=+43.141846709 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-28qt8" (UID: "950cea78-bd24-478c-810c-e398a8075c7d") : secret "cluster-monitoring-operator-tls" not found Jul 03 11:12:56.579485 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:12:56.579227 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls podName:ae916522-5d55-4fbf-b304-a7f05fae5691 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:00.579211646 +0000 UTC m=+43.141887135 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls") pod "dns-default-bfkkq" (UID: "ae916522-5d55-4fbf-b304-a7f05fae5691") : secret "dns-default-metrics-tls" not found Jul 03 11:13:00.516481 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:00.516418 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:00.516515 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:00.516579 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.516593 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:00.516608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.516619 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846768dbb5-2blkk: secret "image-registry-tls" not found Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.516677 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.516688 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls podName:db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:08.516666382 +0000 UTC m=+51.079341889 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls") pod "image-registry-846768dbb5-2blkk" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11") : secret "image-registry-tls" not found Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.516726 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls podName:ed2f81ba-9422-4403-b06e-fcda340b8afa nodeName:}" failed. No retries permitted until 2026-07-03 11:13:08.516712562 +0000 UTC m=+51.079388054 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hz5wv" (UID: "ed2f81ba-9422-4403-b06e-fcda340b8afa") : secret "samples-operator-tls" not found Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.516752 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.516768 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:08.516750344 +0000 UTC m=+51.079425849 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : configmap references non-existent config key: service-ca.crt Jul 03 11:13:00.517045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.516808 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:08.516793688 +0000 UTC m=+51.079469179 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : secret "router-metrics-certs-default" not found Jul 03 11:13:00.617331 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:00.617291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:13:00.617331 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:00.617330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:13:00.617573 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.617473 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 03 11:13:00.617573 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.617473 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 11:13:00.617573 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.617536 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls podName:950cea78-bd24-478c-810c-e398a8075c7d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:08.617518527 +0000 UTC m=+51.180194037 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-28qt8" (UID: "950cea78-bd24-478c-810c-e398a8075c7d") : secret "cluster-monitoring-operator-tls" not found Jul 03 11:13:00.617573 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.617552 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls podName:ae916522-5d55-4fbf-b304-a7f05fae5691 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:08.617543867 +0000 UTC m=+51.180219357 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls") pod "dns-default-bfkkq" (UID: "ae916522-5d55-4fbf-b304-a7f05fae5691") : secret "dns-default-metrics-tls" not found Jul 03 11:13:00.617768 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:00.617639 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:13:00.617818 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.617781 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 03 11:13:00.617870 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:00.617835 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert podName:c9dd690e-80aa-4886-963a-5616b8041655 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:08.617818053 +0000 UTC m=+51.180493572 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rcsx9" (UID: "c9dd690e-80aa-4886-963a-5616b8041655") : secret "networking-console-plugin-cert" not found Jul 03 11:13:03.410038 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.409998 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg" event={"ID":"e7554f43-afeb-4d23-9530-589db05eeb2f","Type":"ContainerStarted","Data":"7ea43acb016e85fcca1f41d624798fc960aafeee09ce78e8a97c346790b29aaf"} Jul 03 11:13:03.414053 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.414029 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" event={"ID":"1c73b128-ad8e-46c4-86bf-1596700247a4","Type":"ContainerStarted","Data":"15c2744d932f58f75ceb7e9ff916c9da0f97f7943b67185e44c122bdc31b6941"} Jul 03 11:13:03.414900 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.414727 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:13:03.416474 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.416171 2581 patch_prober.go:28] interesting pod/console-operator-68b757fb67-p2bsb container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" start-of-body= Jul 03 11:13:03.416474 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.416222 2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" podUID="1c73b128-ad8e-46c4-86bf-1596700247a4" containerName="console-operator" probeResult="failure" output="Get \"https://10.132.0.12:8443/readyz\": dial tcp 10.132.0.12:8443: connect: connection refused" Jul 03 11:13:03.417368 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.417031 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" event={"ID":"c1477b24-cb02-4165-925e-8146aa836065","Type":"ContainerStarted","Data":"a465ec77c6ddf7a3f00c32d16d861b9824e8cd6cac3f6f2f6d56adbdec1bb576"} Jul 03 11:13:03.420018 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.419994 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" event={"ID":"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7","Type":"ContainerStarted","Data":"b1dde5dda9ff06fb32280f621897d8cc30d543b347046203a7e786cccd72b90d"} Jul 03 11:13:03.428811 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.428764 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-5d456fdbcd-2j6rg" podStartSLOduration=18.552154366 podStartE2EDuration="28.428751035s" totalImagesPullingTime="9.876596669s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:03.427988443 +0000 UTC m=+45.990663955" watchObservedRunningTime="2026-07-03 11:13:03.428751035 +0000 UTC m=+45.991426547" Jul 03 11:13:03.451116 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.446939 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" podStartSLOduration=18.629382405 podStartE2EDuration="28.446919871s" totalImagesPullingTime="9.817537466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:03.446569937 +0000 UTC m=+46.009245450" watchObservedRunningTime="2026-07-03 11:13:03.446919871 +0000 UTC m=+46.009595383" Jul 03 11:13:03.467734 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:03.467318 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" podStartSLOduration=18.599706813 podStartE2EDuration="28.467301951s" totalImagesPullingTime="9.867595138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:03.4672591 +0000 UTC m=+46.029934613" watchObservedRunningTime="2026-07-03 11:13:03.467301951 +0000 UTC m=+46.029977463" Jul 03 11:13:04.123459 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.120654 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" podStartSLOduration=19.304429964 podStartE2EDuration="29.120632312s" totalImagesPullingTime="9.816202348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:03.493144113 +0000 UTC m=+46.055819626" watchObservedRunningTime="2026-07-03 11:13:04.120632312 +0000 UTC m=+46.683307806" Jul 03 11:13:04.123459 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.122157 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qhkvs"] Jul 03 11:13:04.140706 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.140667 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.145640 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.145612 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhkvs"] Jul 03 11:13:04.149993 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.149958 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d5hq9\"" Jul 03 11:13:04.150808 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.150262 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Jul 03 11:13:04.150808 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.150574 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Jul 03 11:13:04.250253 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.250214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6d89b76a-0f00-4802-999b-614c7a0ac325-crio-socket\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.250462 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.250294 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.250536 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.250509 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.250594 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.250562 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d89b76a-0f00-4802-999b-614c7a0ac325-data-volume\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.250646 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.250615 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkm7h\" (UniqueName: \"kubernetes.io/projected/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-api-access-wkm7h\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.351892 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.351853 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.352052 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.351915 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d89b76a-0f00-4802-999b-614c7a0ac325-data-volume\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.352052 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.351957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkm7h\" (UniqueName: \"kubernetes.io/projected/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-api-access-wkm7h\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.352052 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.352008 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6d89b76a-0f00-4802-999b-614c7a0ac325-crio-socket\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.352235 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:04.352047 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 11:13:04.352235 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.352066 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.352235 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:04.352120 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:04.852098774 +0000 UTC m=+47.414774270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : secret "insights-runtime-extractor-tls" not found Jul 03 11:13:04.352235 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:04.352152 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 11:13:04.352235 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:04.352204 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:04.852187734 +0000 UTC m=+47.414863246 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : configmap "kube-rbac-proxy" not found Jul 03 11:13:04.352508 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.352255 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6d89b76a-0f00-4802-999b-614c7a0ac325-data-volume\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.352508 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.352346 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6d89b76a-0f00-4802-999b-614c7a0ac325-crio-socket\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.361156 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.361128 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkm7h\" (UniqueName: \"kubernetes.io/projected/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-api-access-wkm7h\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.424497 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.424465 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-rzn28" event={"ID":"c5c70232-5339-4786-bf98-86283e335ea1","Type":"ContainerStarted","Data":"09b800abb346e21bc6b404d861e9aeed2f912ca8776ded72ccc47c205ca443e1"} Jul 03 11:13:04.426128 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.426101 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj" event={"ID":"06f56a11-04a4-4b4d-8cfe-7842af7f0bfe","Type":"ContainerStarted","Data":"f2aa941a243e7f90a5d9be0a8bf09d483f5cb778074e87cdf695cdfc76b50c1f"} Jul 03 11:13:04.429085 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.429060 2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc69354-77c5-484c-b655-7ff694e5e9d9" containerID="07f194c1edb5b0ab18eef4b1e7c69abb2d1823d5009eb087db4be357aeff8445" exitCode=0 Jul 03 11:13:04.429186 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.429139 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tn46" event={"ID":"5dc69354-77c5-484c-b655-7ff694e5e9d9","Type":"ContainerDied","Data":"07f194c1edb5b0ab18eef4b1e7c69abb2d1823d5009eb087db4be357aeff8445"} Jul 03 11:13:04.435304 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.435287 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/0.log" Jul 03 11:13:04.435463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.435322 2581 generic.go:356] "Generic (PLEG): container finished" podID="1c73b128-ad8e-46c4-86bf-1596700247a4" containerID="15c2744d932f58f75ceb7e9ff916c9da0f97f7943b67185e44c122bdc31b6941" exitCode=255 Jul 03 11:13:04.435463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.435421 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" event={"ID":"1c73b128-ad8e-46c4-86bf-1596700247a4","Type":"ContainerDied","Data":"15c2744d932f58f75ceb7e9ff916c9da0f97f7943b67185e44c122bdc31b6941"} Jul 03 11:13:04.435628 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.435607 2581 scope.go:117] "RemoveContainer" containerID="15c2744d932f58f75ceb7e9ff916c9da0f97f7943b67185e44c122bdc31b6941" Jul 03 11:13:04.437070 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.437047 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" event={"ID":"9885b117-aabd-414f-be44-5ee5f7c908eb","Type":"ContainerStarted","Data":"501c154df6871852f186685ff020897f78a156da54e583f799b822dcac588eac"} Jul 03 11:13:04.442966 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.442932 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-rzn28" podStartSLOduration=2.65847277 podStartE2EDuration="12.442919673s" totalImagesPullingTime="9.784446903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:04.442214618 +0000 UTC m=+47.004890132" watchObservedRunningTime="2026-07-03 11:13:04.442919673 +0000 UTC m=+47.005595185" Jul 03 11:13:04.487294 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.487238 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" podStartSLOduration=19.813529319 podStartE2EDuration="29.487216537s" totalImagesPullingTime="9.673687218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:04.485823785 +0000 UTC m=+47.048499297" watchObservedRunningTime="2026-07-03 11:13:04.487216537 +0000 UTC m=+47.049892050" Jul 03 11:13:04.557215 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.556692 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-989f8b8dc-wv8nj" podStartSLOduration=19.777341259 podStartE2EDuration="29.556673958s" totalImagesPullingTime="9.779332699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:04.515414042 +0000 UTC m=+47.078089556" watchObservedRunningTime="2026-07-03 11:13:04.556673958 +0000 UTC m=+47.119349471" Jul 03 11:13:04.857584 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.857544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.857784 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:04.857673 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:04.857784 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:04.857677 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 11:13:04.857784 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:04.857750 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:05.857730488 +0000 UTC m=+48.420406004 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : configmap "kube-rbac-proxy" not found Jul 03 11:13:04.857784 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:04.857782 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 11:13:04.858061 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:04.857821 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:05.857810101 +0000 UTC m=+48.420485591 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : secret "insights-runtime-extractor-tls" not found Jul 03 11:13:05.447164 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.447130 2581 generic.go:356] "Generic (PLEG): container finished" podID="5dc69354-77c5-484c-b655-7ff694e5e9d9" containerID="541c1898a767e3331b3903b583e9bd5a160b36daa2fe860fc9f60a2867ecc2e8" exitCode=0 Jul 03 11:13:05.447588 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.447222 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tn46" event={"ID":"5dc69354-77c5-484c-b655-7ff694e5e9d9","Type":"ContainerDied","Data":"541c1898a767e3331b3903b583e9bd5a160b36daa2fe860fc9f60a2867ecc2e8"} Jul 03 11:13:05.448716 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.448697 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:13:05.449141 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.449126 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/0.log" Jul 03 11:13:05.449208 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.449157 2581 generic.go:356] "Generic (PLEG): container finished" podID="1c73b128-ad8e-46c4-86bf-1596700247a4" containerID="8ce09ff0f3f91d89a8f858e0cc98cd19a020a6a66d8893ebd91680885da36714" exitCode=255 Jul 03 11:13:05.449208 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.449185 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" event={"ID":"1c73b128-ad8e-46c4-86bf-1596700247a4","Type":"ContainerDied","Data":"8ce09ff0f3f91d89a8f858e0cc98cd19a020a6a66d8893ebd91680885da36714"} Jul 03 11:13:05.449293 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.449229 2581 scope.go:117] "RemoveContainer" containerID="15c2744d932f58f75ceb7e9ff916c9da0f97f7943b67185e44c122bdc31b6941" Jul 03 11:13:05.449520 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.449497 2581 scope.go:117] "RemoveContainer" containerID="8ce09ff0f3f91d89a8f858e0cc98cd19a020a6a66d8893ebd91680885da36714" Jul 03 11:13:05.449695 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:05.449674 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-p2bsb_openshift-console-operator(1c73b128-ad8e-46c4-86bf-1596700247a4)\"" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" podUID="1c73b128-ad8e-46c4-86bf-1596700247a4" Jul 03 11:13:05.867897 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.867812 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:05.868045 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.867934 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:05.868045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:05.867939 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 11:13:05.868045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:05.868001 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:07.86798666 +0000 UTC m=+50.430662150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : configmap "kube-rbac-proxy" not found Jul 03 11:13:05.868148 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:05.868067 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 11:13:05.868148 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:05.868115 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:07.868103988 +0000 UTC m=+50.430779479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : secret "insights-runtime-extractor-tls" not found Jul 03 11:13:05.906881 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:05.906853 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8d5tj_1a6b54da-ff3a-4ad1-868b-1f06684af304/dns-node-resolver/0.log" Jul 03 11:13:06.456504 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:06.456464 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8tn46" event={"ID":"5dc69354-77c5-484c-b655-7ff694e5e9d9","Type":"ContainerStarted","Data":"dc9b30962665d1eac79b1e01389024142536a90640c71d5e8d76a22f4eaec77f"} Jul 03 11:13:06.457926 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:06.457909 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:13:06.458222 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:06.458206 2581 scope.go:117] "RemoveContainer" containerID="8ce09ff0f3f91d89a8f858e0cc98cd19a020a6a66d8893ebd91680885da36714" Jul 03 11:13:06.458398 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:06.458379 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-p2bsb_openshift-console-operator(1c73b128-ad8e-46c4-86bf-1596700247a4)\"" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" podUID="1c73b128-ad8e-46c4-86bf-1596700247a4" Jul 03 11:13:06.480551 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:06.480503 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8tn46" podStartSLOduration=7.649686062 podStartE2EDuration="48.480488029s" totalImagesPullingTime="40.830801967s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:06.478820595 +0000 UTC m=+49.041496108" watchObservedRunningTime="2026-07-03 11:13:06.480488029 +0000 UTC m=+49.043163541" Jul 03 11:13:07.106478 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:07.106451 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mc48b_b4cfd292-27df-444a-b0af-43067d41ee34/node-ca/0.log" Jul 03 11:13:07.886319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:07.886282 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:07.886772 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:07.886388 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:07.886772 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:07.886457 2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found Jul 03 11:13:07.886772 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:07.886488 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 11:13:07.886772 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:07.886545 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:11.886530993 +0000 UTC m=+54.449206484 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : secret "insights-runtime-extractor-tls" not found Jul 03 11:13:07.886772 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:07.886558 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:11.886551714 +0000 UTC m=+54.449227204 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : configmap "kube-rbac-proxy" not found Jul 03 11:13:07.908395 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:07.908373 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rzn28_c5c70232-5339-4786-bf98-86283e335ea1/serve-healthcheck-canary/0.log" Jul 03 11:13:08.592730 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:08.592691 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:13:08.592917 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:08.592781 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:08.592917 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:08.592803 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:08.592917 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:08.592861 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:13:08.592917 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.592879 2581 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found Jul 03 11:13:08.593117 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.592944 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls podName:ed2f81ba-9422-4403-b06e-fcda340b8afa nodeName:}" failed. No retries permitted until 2026-07-03 11:13:24.592929169 +0000 UTC m=+67.155604661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls") pod "cluster-samples-operator-56fcf86bf-hz5wv" (UID: "ed2f81ba-9422-4403-b06e-fcda340b8afa") : secret "samples-operator-tls" not found Jul 03 11:13:08.593117 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.592959 2581 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found Jul 03 11:13:08.593117 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.592972 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:24.592954523 +0000 UTC m=+67.155630032 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : configmap references non-existent config key: service-ca.crt Jul 03 11:13:08.593117 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.592974 2581 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found Jul 03 11:13:08.593117 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.592991 2581 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-846768dbb5-2blkk: secret "image-registry-tls" not found Jul 03 11:13:08.593117 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.593018 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs podName:6ef1ae29-90df-42ac-8b5d-049e00ffe01d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:24.593008834 +0000 UTC m=+67.155684324 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs") pod "router-default-678cfbb59c-mm6jx" (UID: "6ef1ae29-90df-42ac-8b5d-049e00ffe01d") : secret "router-metrics-certs-default" not found Jul 03 11:13:08.593117 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.593033 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls podName:db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:24.593024181 +0000 UTC m=+67.155699672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls") pod "image-registry-846768dbb5-2blkk" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11") : secret "image-registry-tls" not found Jul 03 11:13:08.693814 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:08.693782 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:13:08.693984 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:08.693820 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:13:08.693984 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:08.693875 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:13:08.693984 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.693947 2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found Jul 03 11:13:08.694083 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.694021 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls podName:ae916522-5d55-4fbf-b304-a7f05fae5691 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:24.694002632 +0000 UTC m=+67.256678142 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls") pod "dns-default-bfkkq" (UID: "ae916522-5d55-4fbf-b304-a7f05fae5691") : secret "dns-default-metrics-tls" not found Jul 03 11:13:08.694083 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.694020 2581 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found Jul 03 11:13:08.694083 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.694028 2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found Jul 03 11:13:08.694083 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.694076 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls podName:950cea78-bd24-478c-810c-e398a8075c7d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:24.694062098 +0000 UTC m=+67.256737590 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-64786d9cbb-28qt8" (UID: "950cea78-bd24-478c-810c-e398a8075c7d") : secret "cluster-monitoring-operator-tls" not found Jul 03 11:13:08.694215 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:08.694093 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert podName:c9dd690e-80aa-4886-963a-5616b8041655 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:24.69408445 +0000 UTC m=+67.256759941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert") pod "networking-console-plugin-6bffdf9778-rcsx9" (UID: "c9dd690e-80aa-4886-963a-5616b8041655") : secret "networking-console-plugin-cert" not found Jul 03 11:13:11.919068 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:11.919030 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:11.919773 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:11.919147 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:11.919773 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:11.919269 2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found Jul 03 11:13:11.919773 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:11.919322 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls podName:6d89b76a-0f00-4802-999b-614c7a0ac325 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:19.919308171 +0000 UTC m=+62.481983662 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qhkvs" (UID: "6d89b76a-0f00-4802-999b-614c7a0ac325") : secret "insights-runtime-extractor-tls" not found Jul 03 11:13:11.919936 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:11.919771 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6d89b76a-0f00-4802-999b-614c7a0ac325-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:13.116633 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:13.116603 2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:13:13.117122 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:13.117081 2581 scope.go:117] "RemoveContainer" containerID="8ce09ff0f3f91d89a8f858e0cc98cd19a020a6a66d8893ebd91680885da36714" Jul 03 11:13:13.117287 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:13.117263 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-p2bsb_openshift-console-operator(1c73b128-ad8e-46c4-86bf-1596700247a4)\"" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" podUID="1c73b128-ad8e-46c4-86bf-1596700247a4" Jul 03 11:13:13.415008 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:13.414969 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:13:13.484224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:13.484192 2581 scope.go:117] "RemoveContainer" containerID="8ce09ff0f3f91d89a8f858e0cc98cd19a020a6a66d8893ebd91680885da36714" Jul 03 11:13:13.484423 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:13.484399 2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-68b757fb67-p2bsb_openshift-console-operator(1c73b128-ad8e-46c4-86bf-1596700247a4)\"" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" podUID="1c73b128-ad8e-46c4-86bf-1596700247a4" Jul 03 11:13:18.344227 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:18.344192 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-47drl" Jul 03 11:13:19.991515 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:19.991483 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:19.993965 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:19.993933 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6d89b76a-0f00-4802-999b-614c7a0ac325-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qhkvs\" (UID: \"6d89b76a-0f00-4802-999b-614c7a0ac325\") " pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:20.056636 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:20.056606 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-d5hq9\"" Jul 03 11:13:20.064282 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:20.064263 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qhkvs" Jul 03 11:13:20.193984 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:20.193909 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qhkvs"] Jul 03 11:13:20.196560 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:20.196533 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d89b76a_0f00_4802_999b_614c7a0ac325.slice/crio-3ec00577611783f417e1a3853ac7ece9474a14b0690afdc094818abc893a09e2 WatchSource:0}: Error finding container 3ec00577611783f417e1a3853ac7ece9474a14b0690afdc094818abc893a09e2: Status 404 returned error can't find the container with id 3ec00577611783f417e1a3853ac7ece9474a14b0690afdc094818abc893a09e2 Jul 03 11:13:20.503933 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:20.503898 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhkvs" event={"ID":"6d89b76a-0f00-4802-999b-614c7a0ac325","Type":"ContainerStarted","Data":"3ec00577611783f417e1a3853ac7ece9474a14b0690afdc094818abc893a09e2"} Jul 03 11:13:22.717771 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.717722 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:13:22.718226 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.717834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:13:22.720316 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.720294 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Jul 03 11:13:22.720463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.720363 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Jul 03 11:13:22.730451 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.730406 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/0b814191-d67d-47b0-8c3f-af739872704b-original-pull-secret\") pod \"global-pull-secret-syncer-q2wrq\" (UID: \"0b814191-d67d-47b0-8c3f-af739872704b\") " pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:13:22.730561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.730463 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6925168-d48c-405d-8a6c-30384df3214c-metrics-certs\") pod \"network-metrics-daemon-dqt7b\" (UID: \"e6925168-d48c-405d-8a6c-30384df3214c\") " pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:13:22.818667 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.818631 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:13:22.821385 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.821341 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2cnmr\" (UniqueName: \"kubernetes.io/projected/69bdf6b0-795e-4c72-b8f6-dd577472c2de-kube-api-access-2cnmr\") pod \"network-check-target-76w8c\" (UID: \"69bdf6b0-795e-4c72-b8f6-dd577472c2de\") " pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:13:22.999836 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:22.999754 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-q2wrq" Jul 03 11:13:23.009375 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:23.009350 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wfk2t\"" Jul 03 11:13:23.013411 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:23.013381 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qx6zp\"" Jul 03 11:13:23.017356 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:23.017336 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:13:23.021403 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:23.021381 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-dqt7b" Jul 03 11:13:23.875076 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:23.875047 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-76w8c"] Jul 03 11:13:23.879562 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:23.879416 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bdf6b0_795e_4c72_b8f6_dd577472c2de.slice/crio-3377cf78b01581bb3414e9bc0096c920b6993a4d1eb2d5c66f9899e6e682189a WatchSource:0}: Error finding container 3377cf78b01581bb3414e9bc0096c920b6993a4d1eb2d5c66f9899e6e682189a: Status 404 returned error can't find the container with id 3377cf78b01581bb3414e9bc0096c920b6993a4d1eb2d5c66f9899e6e682189a Jul 03 11:13:23.890975 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:23.890941 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-dqt7b"] Jul 03 11:13:23.907459 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:23.907397 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-q2wrq"] Jul 03 11:13:23.914554 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:23.914526 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b814191_d67d_47b0_8c3f_af739872704b.slice/crio-72633d3434ff730208ecbefaaed3141115b4af32533725fef84b1cca921b44ed WatchSource:0}: Error finding container 72633d3434ff730208ecbefaaed3141115b4af32533725fef84b1cca921b44ed: Status 404 returned error can't find the container with id 72633d3434ff730208ecbefaaed3141115b4af32533725fef84b1cca921b44ed Jul 03 11:13:23.988904 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:23.988817 2581 scope.go:117] "RemoveContainer" containerID="8ce09ff0f3f91d89a8f858e0cc98cd19a020a6a66d8893ebd91680885da36714" Jul 03 11:13:24.521101 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.521074 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqt7b" event={"ID":"e6925168-d48c-405d-8a6c-30384df3214c","Type":"ContainerStarted","Data":"40ea0ac3ff72c38ea8d73aaba443e582922c180354df5a31807bdebc5b8fa67d"} Jul 03 11:13:24.522931 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.522903 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:13:24.523022 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.522997 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" event={"ID":"1c73b128-ad8e-46c4-86bf-1596700247a4","Type":"ContainerStarted","Data":"849a68bb65d1771fd9f89a26a1c40bc7984025268a3dfcfd268a0b22b902cd87"} Jul 03 11:13:24.523414 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.523377 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:13:24.524730 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.524700 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-76w8c" event={"ID":"69bdf6b0-795e-4c72-b8f6-dd577472c2de","Type":"ContainerStarted","Data":"e7c9305f754400c6452289c1d90af7656121bb77a2b266eb36372b929391157a"} Jul 03 11:13:24.524827 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.524749 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:13:24.524827 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.524820 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-76w8c" event={"ID":"69bdf6b0-795e-4c72-b8f6-dd577472c2de","Type":"ContainerStarted","Data":"3377cf78b01581bb3414e9bc0096c920b6993a4d1eb2d5c66f9899e6e682189a"} Jul 03 11:13:24.525855 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.525833 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2wrq" event={"ID":"0b814191-d67d-47b0-8c3f-af739872704b","Type":"ContainerStarted","Data":"72633d3434ff730208ecbefaaed3141115b4af32533725fef84b1cca921b44ed"} Jul 03 11:13:24.527311 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.527291 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhkvs" event={"ID":"6d89b76a-0f00-4802-999b-614c7a0ac325","Type":"ContainerStarted","Data":"32dbe11cf507c6252de6ae6d357f3e46c80a95093e9f62725ab1f7ec02775415"} Jul 03 11:13:24.553840 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.553793 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-76w8c" podStartSLOduration=66.553776085 podStartE2EDuration="1m6.553776085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:24.55339744 +0000 UTC m=+67.116072954" watchObservedRunningTime="2026-07-03 11:13:24.553776085 +0000 UTC m=+67.116451596" Jul 03 11:13:24.562880 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.562859 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-68b757fb67-p2bsb" Jul 03 11:13:24.636552 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.636511 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:13:24.636758 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.636620 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:24.637265 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.636892 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:24.637449 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.637416 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:13:24.638004 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.637980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-service-ca-bundle\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:24.640359 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.640327 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/ed2f81ba-9422-4403-b06e-fcda340b8afa-samples-operator-tls\") pod \"cluster-samples-operator-56fcf86bf-hz5wv\" (UID: \"ed2f81ba-9422-4403-b06e-fcda340b8afa\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:13:24.645254 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.645157 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ef1ae29-90df-42ac-8b5d-049e00ffe01d-metrics-certs\") pod \"router-default-678cfbb59c-mm6jx\" (UID: \"6ef1ae29-90df-42ac-8b5d-049e00ffe01d\") " pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:24.646554 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.646476 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"image-registry-846768dbb5-2blkk\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:13:24.738983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.738933 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:13:24.739158 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.738993 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:13:24.739158 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.739041 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:13:24.742187 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.742154 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9dd690e-80aa-4886-963a-5616b8041655-networking-console-plugin-cert\") pod \"networking-console-plugin-6bffdf9778-rcsx9\" (UID: \"c9dd690e-80aa-4886-963a-5616b8041655\") " pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:13:24.742371 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.742319 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ae916522-5d55-4fbf-b304-a7f05fae5691-metrics-tls\") pod \"dns-default-bfkkq\" (UID: \"ae916522-5d55-4fbf-b304-a7f05fae5691\") " pod="openshift-dns/dns-default-bfkkq" Jul 03 11:13:24.744224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.744198 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/950cea78-bd24-478c-810c-e398a8075c7d-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-64786d9cbb-28qt8\" (UID: \"950cea78-bd24-478c-810c-e398a8075c7d\") " pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:13:24.848872 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.848782 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-58shg\"" Jul 03 11:13:24.856984 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.856946 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:13:24.876617 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.876380 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-czfxb\"" Jul 03 11:13:24.884802 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.884774 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" Jul 03 11:13:24.899410 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.899382 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-lxj4p\"" Jul 03 11:13:24.907217 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.907186 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:24.923899 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.923866 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-qvzh2\"" Jul 03 11:13:24.932470 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.932070 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" Jul 03 11:13:24.939085 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.939053 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-9wzc8\"" Jul 03 11:13:24.946698 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.946423 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" Jul 03 11:13:24.958752 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.958723 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kggc2\"" Jul 03 11:13:24.967089 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:24.966554 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bfkkq" Jul 03 11:13:25.096611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.094840 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-846768dbb5-2blkk"] Jul 03 11:13:25.120199 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.120116 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv"] Jul 03 11:13:25.139545 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.139505 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-678cfbb59c-mm6jx"] Jul 03 11:13:25.331344 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:25.331089 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb37c72c_8cdf_4aff_a2dc_59d6a9f4eb11.slice/crio-c343e34cfad090445f3e76fa47bfee9d37170bc2e6113f8f5c173cd75c72475e WatchSource:0}: Error finding container c343e34cfad090445f3e76fa47bfee9d37170bc2e6113f8f5c173cd75c72475e: Status 404 returned error can't find the container with id c343e34cfad090445f3e76fa47bfee9d37170bc2e6113f8f5c173cd75c72475e Jul 03 11:13:25.333768 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:25.333734 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef1ae29_90df_42ac_8b5d_049e00ffe01d.slice/crio-3226d8b1755a815a1f4463a3806fd4c3a23b7661090a5c47d3f3086c8411c929 WatchSource:0}: Error finding container 3226d8b1755a815a1f4463a3806fd4c3a23b7661090a5c47d3f3086c8411c929: Status 404 returned error can't find the container with id 3226d8b1755a815a1f4463a3806fd4c3a23b7661090a5c47d3f3086c8411c929 Jul 03 11:13:25.539782 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.538829 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" event={"ID":"6ef1ae29-90df-42ac-8b5d-049e00ffe01d","Type":"ContainerStarted","Data":"8c654ffebe8f33b22a92bd9e6f09a911afba4d5eebd33afec2fcc42a17bf553f"} Jul 03 11:13:25.539782 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.538896 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" event={"ID":"6ef1ae29-90df-42ac-8b5d-049e00ffe01d","Type":"ContainerStarted","Data":"3226d8b1755a815a1f4463a3806fd4c3a23b7661090a5c47d3f3086c8411c929"} Jul 03 11:13:25.543854 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.543700 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" event={"ID":"ed2f81ba-9422-4403-b06e-fcda340b8afa","Type":"ContainerStarted","Data":"dbcde398b8b801447095f1324fc2e0dc190ad2ed77b027b0d10e9c4211e6a876"} Jul 03 11:13:25.552543 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.552357 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhkvs" event={"ID":"6d89b76a-0f00-4802-999b-614c7a0ac325","Type":"ContainerStarted","Data":"aee538b4f773f4f6b30b824194ba1bd3d8a01a4d13a2b5d7325a253a1e980410"} Jul 03 11:13:25.555951 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.555745 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9"] Jul 03 11:13:25.562214 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.562184 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" event={"ID":"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11","Type":"ContainerStarted","Data":"545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743"} Jul 03 11:13:25.562299 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.562241 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" event={"ID":"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11","Type":"ContainerStarted","Data":"c343e34cfad090445f3e76fa47bfee9d37170bc2e6113f8f5c173cd75c72475e"} Jul 03 11:13:25.562632 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.562602 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:13:25.565068 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.564914 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" podStartSLOduration=50.564893029 podStartE2EDuration="50.564893029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:25.563408763 +0000 UTC m=+68.126084275" watchObservedRunningTime="2026-07-03 11:13:25.564893029 +0000 UTC m=+68.127568541" Jul 03 11:13:25.566275 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:25.566137 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9dd690e_80aa_4886_963a_5616b8041655.slice/crio-86abe4be015c4b8e5338e5544035a1a1b8a5095c56152e32cfe92a47323e6e69 WatchSource:0}: Error finding container 86abe4be015c4b8e5338e5544035a1a1b8a5095c56152e32cfe92a47323e6e69: Status 404 returned error can't find the container with id 86abe4be015c4b8e5338e5544035a1a1b8a5095c56152e32cfe92a47323e6e69 Jul 03 11:13:25.575004 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.574970 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bfkkq"] Jul 03 11:13:25.579695 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:25.579657 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae916522_5d55_4fbf_b304_a7f05fae5691.slice/crio-ef563f0d3061a0408d6d45c1fcda72416d66d53f8d724605fd8f161b5e08117f WatchSource:0}: Error finding container ef563f0d3061a0408d6d45c1fcda72416d66d53f8d724605fd8f161b5e08117f: Status 404 returned error can't find the container with id ef563f0d3061a0408d6d45c1fcda72416d66d53f8d724605fd8f161b5e08117f Jul 03 11:13:25.593704 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.593616 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" podStartSLOduration=67.593594969 podStartE2EDuration="1m7.593594969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:25.593006298 +0000 UTC m=+68.155681811" watchObservedRunningTime="2026-07-03 11:13:25.593594969 +0000 UTC m=+68.156270483" Jul 03 11:13:25.600028 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.599959 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8"] Jul 03 11:13:25.602590 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:25.601902 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod950cea78_bd24_478c_810c_e398a8075c7d.slice/crio-5ac44819b8ab9c1cd8b076338b1f50a06de6bac745f8146b12bce5e2cd46bca8 WatchSource:0}: Error finding container 5ac44819b8ab9c1cd8b076338b1f50a06de6bac745f8146b12bce5e2cd46bca8: Status 404 returned error can't find the container with id 5ac44819b8ab9c1cd8b076338b1f50a06de6bac745f8146b12bce5e2cd46bca8 Jul 03 11:13:25.917532 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.917498 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:25.921289 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:25.921067 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:26.569686 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:26.569598 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfkkq" event={"ID":"ae916522-5d55-4fbf-b304-a7f05fae5691","Type":"ContainerStarted","Data":"ef563f0d3061a0408d6d45c1fcda72416d66d53f8d724605fd8f161b5e08117f"} Jul 03 11:13:26.571611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:26.571538 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" event={"ID":"950cea78-bd24-478c-810c-e398a8075c7d","Type":"ContainerStarted","Data":"5ac44819b8ab9c1cd8b076338b1f50a06de6bac745f8146b12bce5e2cd46bca8"} Jul 03 11:13:26.573056 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:26.572990 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" event={"ID":"c9dd690e-80aa-4886-963a-5616b8041655","Type":"ContainerStarted","Data":"86abe4be015c4b8e5338e5544035a1a1b8a5095c56152e32cfe92a47323e6e69"} Jul 03 11:13:26.577486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:26.576475 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqt7b" event={"ID":"e6925168-d48c-405d-8a6c-30384df3214c","Type":"ContainerStarted","Data":"297e2cb2465f5be9e1f7df8541fbddf8ee0a86de9188414b7f98bc2a5ee752a3"} Jul 03 11:13:26.577486 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:26.576522 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-dqt7b" event={"ID":"e6925168-d48c-405d-8a6c-30384df3214c","Type":"ContainerStarted","Data":"4e0442bb42802b98a001e7fa0a395c06046f250da19637655cfa2b82aeee4b79"} Jul 03 11:13:26.577847 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:26.577784 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:26.579752 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:26.579642 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-678cfbb59c-mm6jx" Jul 03 11:13:26.594831 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:26.594584 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-dqt7b" podStartSLOduration=67.093424851 podStartE2EDuration="1m8.59456074s" totalImagesPullingTime="1.501135889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:26.593861289 +0000 UTC m=+69.156536802" watchObservedRunningTime="2026-07-03 11:13:26.59456074 +0000 UTC m=+69.157236252" Jul 03 11:13:32.602583 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.602476 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" event={"ID":"ed2f81ba-9422-4403-b06e-fcda340b8afa","Type":"ContainerStarted","Data":"0da0c93470c7ca5aa5f46f2b6b947247b64dc6dcc616f7918a0570dd810d7a56"} Jul 03 11:13:32.602583 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.602540 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" event={"ID":"ed2f81ba-9422-4403-b06e-fcda340b8afa","Type":"ContainerStarted","Data":"debdec42389a265f61959111c4cca5056b2bda1a5bb3d527efc98f36288aa86e"} Jul 03 11:13:32.610094 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.609786 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfkkq" event={"ID":"ae916522-5d55-4fbf-b304-a7f05fae5691","Type":"ContainerStarted","Data":"61ab99602dfbb4779f9149512c77fe98d952f9226432babca96328c18122db36"} Jul 03 11:13:32.627343 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.627307 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" event={"ID":"950cea78-bd24-478c-810c-e398a8075c7d","Type":"ContainerStarted","Data":"fa4131041434dc5923d72c3847e930222abdfdf636fca30ff8d6494bdafad2bc"} Jul 03 11:13:32.629352 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.629028 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-56fcf86bf-hz5wv" podStartSLOduration=50.757360773 podStartE2EDuration="57.629010045s" totalImagesPullingTime="6.871649272s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:32.628670348 +0000 UTC m=+75.191345861" watchObservedRunningTime="2026-07-03 11:13:32.629010045 +0000 UTC m=+75.191685559" Jul 03 11:13:32.635066 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.635037 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-q2wrq" event={"ID":"0b814191-d67d-47b0-8c3f-af739872704b","Type":"ContainerStarted","Data":"9ad608303d30ffddd93a4dd1794d96c5d3498635138e79980592c2ee34db27fc"} Jul 03 11:13:32.641307 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.640723 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qhkvs" event={"ID":"6d89b76a-0f00-4802-999b-614c7a0ac325","Type":"ContainerStarted","Data":"3b757259208424ac5f3de5cd5d770ccf34107659f29712c8cf410371d160a04c"} Jul 03 11:13:32.643231 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.643202 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" event={"ID":"c9dd690e-80aa-4886-963a-5616b8041655","Type":"ContainerStarted","Data":"19db2e368aece7bab8d224f478af8d264734d354c9bb425fe3790223c870c956"} Jul 03 11:13:32.651500 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.651424 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-64786d9cbb-28qt8" podStartSLOduration=50.985199526 podStartE2EDuration="57.651403471s" totalImagesPullingTime="6.666203945s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:32.649111488 +0000 UTC m=+75.211787001" watchObservedRunningTime="2026-07-03 11:13:32.651403471 +0000 UTC m=+75.214078984" Jul 03 11:13:32.668383 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.668330 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-q2wrq" podStartSLOduration=66.315140335 podStartE2EDuration="1m14.668308138s" totalImagesPullingTime="8.353167803s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:12:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:32.667847325 +0000 UTC m=+75.230522838" watchObservedRunningTime="2026-07-03 11:13:32.668308138 +0000 UTC m=+75.230983650" Jul 03 11:13:32.689803 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.689761 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6bffdf9778-rcsx9" podStartSLOduration=33.988200996 podStartE2EDuration="40.68974436s" totalImagesPullingTime="6.701543364s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:32.688762021 +0000 UTC m=+75.251437534" watchObservedRunningTime="2026-07-03 11:13:32.68974436 +0000 UTC m=+75.252419872" Jul 03 11:13:32.976172 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.976112 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qhkvs" podStartSLOduration=17.057511491 podStartE2EDuration="28.976092371s" totalImagesPullingTime="11.91858088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:13:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:32.714370478 +0000 UTC m=+75.277045992" watchObservedRunningTime="2026-07-03 11:13:32.976092371 +0000 UTC m=+75.538767883" Jul 03 11:13:32.976880 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.976861 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46"] Jul 03 11:13:32.998179 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.998140 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" Jul 03 11:13:32.998360 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:32.998127 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46"] Jul 03 11:13:33.000663 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.000639 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" Jul 03 11:13:33.000800 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.000779 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-dsddb\"" Jul 03 11:13:33.116091 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.116051 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4a585b5-9cc0-494d-bf0e-025e2e95c227-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9px46\" (UID: \"a4a585b5-9cc0-494d-bf0e-025e2e95c227\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" Jul 03 11:13:33.217522 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.217478 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4a585b5-9cc0-494d-bf0e-025e2e95c227-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9px46\" (UID: \"a4a585b5-9cc0-494d-bf0e-025e2e95c227\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" Jul 03 11:13:33.217707 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:33.217628 2581 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found Jul 03 11:13:33.217707 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:33.217697 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4a585b5-9cc0-494d-bf0e-025e2e95c227-tls-certificates podName:a4a585b5-9cc0-494d-bf0e-025e2e95c227 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:33.717681949 +0000 UTC m=+76.280357439 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/a4a585b5-9cc0-494d-bf0e-025e2e95c227-tls-certificates") pod "prometheus-operator-admission-webhook-d559d66d5-9px46" (UID: "a4a585b5-9cc0-494d-bf0e-025e2e95c227") : secret "prometheus-operator-admission-webhook-tls" not found Jul 03 11:13:33.649498 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.649459 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bfkkq" event={"ID":"ae916522-5d55-4fbf-b304-a7f05fae5691","Type":"ContainerStarted","Data":"55d14444cb4adb26af1ca18736d3603a61693186720816656a9cd34ef7946db3"} Jul 03 11:13:33.680133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.680086 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bfkkq" podStartSLOduration=34.984927965 podStartE2EDuration="41.680069579s" totalImagesPullingTime="6.695141614s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:12:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:33.678820749 +0000 UTC m=+76.241496262" watchObservedRunningTime="2026-07-03 11:13:33.680069579 +0000 UTC m=+76.242745090" Jul 03 11:13:33.721192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.721146 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4a585b5-9cc0-494d-bf0e-025e2e95c227-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9px46\" (UID: \"a4a585b5-9cc0-494d-bf0e-025e2e95c227\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" Jul 03 11:13:33.724086 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.724049 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/a4a585b5-9cc0-494d-bf0e-025e2e95c227-tls-certificates\") pod \"prometheus-operator-admission-webhook-d559d66d5-9px46\" (UID: \"a4a585b5-9cc0-494d-bf0e-025e2e95c227\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" Jul 03 11:13:33.907693 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:33.907608 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" Jul 03 11:13:34.077280 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:34.077243 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46"] Jul 03 11:13:34.080085 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:34.080048 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4a585b5_9cc0_494d_bf0e_025e2e95c227.slice/crio-2820f749a89b47edcbed58e071e93b98ca7f05b36aa1a5125f651ff392f14c13 WatchSource:0}: Error finding container 2820f749a89b47edcbed58e071e93b98ca7f05b36aa1a5125f651ff392f14c13: Status 404 returned error can't find the container with id 2820f749a89b47edcbed58e071e93b98ca7f05b36aa1a5125f651ff392f14c13 Jul 03 11:13:34.653962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:34.653923 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" event={"ID":"a4a585b5-9cc0-494d-bf0e-025e2e95c227","Type":"ContainerStarted","Data":"2820f749a89b47edcbed58e071e93b98ca7f05b36aa1a5125f651ff392f14c13"} Jul 03 11:13:34.654325 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:34.654156 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bfkkq" Jul 03 11:13:36.663953 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:36.663918 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" event={"ID":"a4a585b5-9cc0-494d-bf0e-025e2e95c227","Type":"ContainerStarted","Data":"a5a2b047e4dab5e405f13d6d1d148caa40495e06b04ff05468b68e23a342bada"} Jul 03 11:13:36.664460 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:36.664129 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" Jul 03 11:13:36.668990 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:36.668963 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" Jul 03 11:13:36.680282 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:36.680234 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-d559d66d5-9px46" podStartSLOduration=3.028018989 podStartE2EDuration="4.68021993s" totalImagesPullingTime="1.652200941s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:36.68002922 +0000 UTC m=+79.242704732" watchObservedRunningTime="2026-07-03 11:13:36.68021993 +0000 UTC m=+79.242895442" Jul 03 11:13:43.400361 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.400248 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97"] Jul 03 11:13:43.403443 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.403411 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.405414 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.405392 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm"] Jul 03 11:13:43.407498 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.407473 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.408200 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.408178 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Jul 03 11:13:43.408270 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.408239 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d579t\"" Jul 03 11:13:43.408324 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.408306 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Jul 03 11:13:43.408887 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.408872 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Jul 03 11:13:43.410575 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.410541 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Jul 03 11:13:43.411209 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.411100 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Jul 03 11:13:43.411274 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.411175 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Jul 03 11:13:43.412126 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.411977 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-wzv8d\"" Jul 03 11:13:43.420812 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.420787 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97"] Jul 03 11:13:43.423522 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.423502 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kkdgv"] Jul 03 11:13:43.425657 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.425640 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.429076 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.428863 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Jul 03 11:13:43.429305 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.429288 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm"] Jul 03 11:13:43.429491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.429473 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-s5r7q\"" Jul 03 11:13:43.429577 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.429512 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Jul 03 11:13:43.430957 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.430936 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Jul 03 11:13:43.498888 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.498857 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-textfile\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.498888 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.498890 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.499133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.498912 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-root\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.499133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.498963 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-tls\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.499133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499007 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.499133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499033 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6bcf17a-1e79-4923-b249-6370a485725d-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.499133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499080 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.499133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499108 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-metrics-client-ca\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.499345 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499148 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bl8q2\" (UniqueName: \"kubernetes.io/projected/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-kube-api-access-bl8q2\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.499345 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499189 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdnp8\" (UniqueName: \"kubernetes.io/projected/c6bcf17a-1e79-4923-b249-6370a485725d-kube-api-access-rdnp8\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.499345 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499221 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.499345 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499288 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.499345 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499316 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6bcf17a-1e79-4923-b249-6370a485725d-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.499345 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499338 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.499558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499357 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.499558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499390 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqbsh\" (UniqueName: \"kubernetes.io/projected/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-kube-api-access-hqbsh\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.499558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-sys\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.499558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499463 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-wtmp\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.499558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.499487 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.600221 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600180 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.600463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600232 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.600463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600250 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6bcf17a-1e79-4923-b249-6370a485725d-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.600463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600268 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.600463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600291 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600463 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:43.600310 2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 03 11:13:43.600463 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:43.600390 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls podName:c6bcf17a-1e79-4923-b249-6370a485725d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:44.100367199 +0000 UTC m=+86.663042706 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-fd7tm" (UID: "c6bcf17a-1e79-4923-b249-6370a485725d") : secret "kube-state-metrics-tls" not found Jul 03 11:13:43.600463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600319 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqbsh\" (UniqueName: \"kubernetes.io/projected/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-kube-api-access-hqbsh\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600491 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-sys\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600521 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-wtmp\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600566 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600593 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-sys\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600604 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-textfile\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600650 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600687 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-root\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600717 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-tls\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600745 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600767 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6bcf17a-1e79-4923-b249-6370a485725d-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600769 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6bcf17a-1e79-4923-b249-6370a485725d-volume-directive-shadow\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600783 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-wtmp\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600796 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.600853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600823 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-metrics-client-ca\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600861 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-textfile\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bl8q2\" (UniqueName: \"kubernetes.io/projected/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-kube-api-access-bl8q2\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:43.600868 2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.600923 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-root\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:43.600940 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls podName:d32ab786-7fe5-4249-97b1-b4476fc7d7c0 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:44.100923195 +0000 UTC m=+86.663598687 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-pld97" (UID: "d32ab786-7fe5-4249-97b1-b4476fc7d7c0") : secret "openshift-state-metrics-tls" not found Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:43.600875 2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.601012 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.601085 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdnp8\" (UniqueName: \"kubernetes.io/projected/c6bcf17a-1e79-4923-b249-6370a485725d-kube-api-access-rdnp8\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.601145 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:43.601220 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-tls podName:61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc nodeName:}" failed. No retries permitted until 2026-07-03 11:13:44.101206717 +0000 UTC m=+86.663882211 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-tls") pod "node-exporter-kkdgv" (UID: "61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc") : secret "node-exporter-tls" not found Jul 03 11:13:43.601561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.601478 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-metrics-client-ca\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.601956 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.601639 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-metrics-client-ca\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.601956 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.601871 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6bcf17a-1e79-4923-b249-6370a485725d-metrics-client-ca\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.603260 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.603239 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.603689 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.603666 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:43.603896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.603868 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.611733 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.611700 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqbsh\" (UniqueName: \"kubernetes.io/projected/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-kube-api-access-hqbsh\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:43.612770 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.612739 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdnp8\" (UniqueName: \"kubernetes.io/projected/c6bcf17a-1e79-4923-b249-6370a485725d-kube-api-access-rdnp8\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:43.612770 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:43.612769 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bl8q2\" (UniqueName: \"kubernetes.io/projected/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-kube-api-access-bl8q2\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:44.105706 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.105670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:44.105888 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.105734 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:44.105888 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.105759 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-tls\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:44.105888 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:44.105843 2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found Jul 03 11:13:44.105888 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:44.105869 2581 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found Jul 03 11:13:44.106045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:44.105915 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls podName:c6bcf17a-1e79-4923-b249-6370a485725d nodeName:}" failed. No retries permitted until 2026-07-03 11:13:45.105896859 +0000 UTC m=+87.668572348 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls") pod "kube-state-metrics-7c9dbb8b7d-fd7tm" (UID: "c6bcf17a-1e79-4923-b249-6370a485725d") : secret "kube-state-metrics-tls" not found Jul 03 11:13:44.106045 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:13:44.105939 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls podName:d32ab786-7fe5-4249-97b1-b4476fc7d7c0 nodeName:}" failed. No retries permitted until 2026-07-03 11:13:45.105923238 +0000 UTC m=+87.668598746 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls") pod "openshift-state-metrics-54cf96dfcf-pld97" (UID: "d32ab786-7fe5-4249-97b1-b4476fc7d7c0") : secret "openshift-state-metrics-tls" not found Jul 03 11:13:44.108111 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.108092 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc-node-exporter-tls\") pod \"node-exporter-kkdgv\" (UID: \"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc\") " pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:44.336018 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.335982 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kkdgv" Jul 03 11:13:44.346094 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:44.346063 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61bfe97f_6da4_4ace_b0bb_79f5a72ac2fc.slice/crio-85cbe9d7918cf82da49715cd7f70a7df12c5fa986475d0a63423d863ae7012bf WatchSource:0}: Error finding container 85cbe9d7918cf82da49715cd7f70a7df12c5fa986475d0a63423d863ae7012bf: Status 404 returned error can't find the container with id 85cbe9d7918cf82da49715cd7f70a7df12c5fa986475d0a63423d863ae7012bf Jul 03 11:13:44.659215 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.659185 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bfkkq" Jul 03 11:13:44.696123 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.696088 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkdgv" event={"ID":"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc","Type":"ContainerStarted","Data":"85cbe9d7918cf82da49715cd7f70a7df12c5fa986475d0a63423d863ae7012bf"} Jul 03 11:13:44.702775 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.702744 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:13:44.706057 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.706033 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.711177 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.711140 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 03 11:13:44.711177 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.711168 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 03 11:13:44.711407 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.711256 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 03 11:13:44.711527 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.711499 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 03 11:13:44.711661 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.711636 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 03 11:13:44.711742 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.711677 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 03 11:13:44.711742 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.711719 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 03 11:13:44.711879 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.711677 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 03 11:13:44.712035 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.712020 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 03 11:13:44.712186 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.712157 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pbnqg\"" Jul 03 11:13:44.729946 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.729919 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:13:44.813515 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813467 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.813697 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813566 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.813697 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813599 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.813697 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.813869 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813743 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tks7\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-kube-api-access-6tks7\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.813869 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813799 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.813951 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813888 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.813951 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813909 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-web-config\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.814039 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.813961 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.814091 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.814036 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.814091 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.814079 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-config-out\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.814191 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.814113 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.814255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.814201 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915076 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.914983 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915076 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915029 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tks7\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-kube-api-access-6tks7\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915076 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915051 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915081 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915101 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-web-config\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915151 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915181 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915203 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-config-out\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915274 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915344 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.915800 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.915367 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.916361 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.916328 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.916595 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.916221 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.917173 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.917145 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.919796 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.919767 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-web-config\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.920016 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.919994 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-config-out\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.920495 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.920473 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.921047 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.920979 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.921168 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.921119 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.921243 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.921223 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-tls-assets\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.921505 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.921478 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-config-volume\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.921625 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.921528 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.921686 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.921662 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:44.926120 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:44.926099 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tks7\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-kube-api-access-6tks7\") pod \"alertmanager-main-0\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:45.018085 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.018047 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:13:45.117037 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.117007 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:45.117202 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.117065 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:45.119964 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.119929 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/d32ab786-7fe5-4249-97b1-b4476fc7d7c0-openshift-state-metrics-tls\") pod \"openshift-state-metrics-54cf96dfcf-pld97\" (UID: \"d32ab786-7fe5-4249-97b1-b4476fc7d7c0\") " pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:45.120571 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.120526 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6bcf17a-1e79-4923-b249-6370a485725d-kube-state-metrics-tls\") pod \"kube-state-metrics-7c9dbb8b7d-fd7tm\" (UID: \"c6bcf17a-1e79-4923-b249-6370a485725d\") " pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:45.180314 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.180183 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:13:45.185182 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:45.185153 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3d062e1_3c80_434c_bd10_df89727e48fb.slice/crio-8441f2afe6c66ebf347dacaf7688bd78928ec5f8c404f0f6806ec5db17ce6724 WatchSource:0}: Error finding container 8441f2afe6c66ebf347dacaf7688bd78928ec5f8c404f0f6806ec5db17ce6724: Status 404 returned error can't find the container with id 8441f2afe6c66ebf347dacaf7688bd78928ec5f8c404f0f6806ec5db17ce6724 Jul 03 11:13:45.218000 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.217930 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" Jul 03 11:13:45.224074 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.224053 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" Jul 03 11:13:45.389012 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.388983 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97"] Jul 03 11:13:45.391860 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:45.391831 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd32ab786_7fe5_4249_97b1_b4476fc7d7c0.slice/crio-06fa281bc0328d55af9a682fa6f659d203f98d386d83b27a8c0f47b6085e2f1e WatchSource:0}: Error finding container 06fa281bc0328d55af9a682fa6f659d203f98d386d83b27a8c0f47b6085e2f1e: Status 404 returned error can't find the container with id 06fa281bc0328d55af9a682fa6f659d203f98d386d83b27a8c0f47b6085e2f1e Jul 03 11:13:45.407978 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.407897 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm"] Jul 03 11:13:45.411366 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:45.411337 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6bcf17a_1e79_4923_b249_6370a485725d.slice/crio-61eb463e11cf546d10e8c8afa4b46abd0adbc86b88bf4f94c2a7c7e0465d9e41 WatchSource:0}: Error finding container 61eb463e11cf546d10e8c8afa4b46abd0adbc86b88bf4f94c2a7c7e0465d9e41: Status 404 returned error can't find the container with id 61eb463e11cf546d10e8c8afa4b46abd0adbc86b88bf4f94c2a7c7e0465d9e41 Jul 03 11:13:45.702513 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.702411 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" event={"ID":"c6bcf17a-1e79-4923-b249-6370a485725d","Type":"ContainerStarted","Data":"61eb463e11cf546d10e8c8afa4b46abd0adbc86b88bf4f94c2a7c7e0465d9e41"} Jul 03 11:13:45.703718 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.703692 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerStarted","Data":"8441f2afe6c66ebf347dacaf7688bd78928ec5f8c404f0f6806ec5db17ce6724"} Jul 03 11:13:45.706015 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.705990 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" event={"ID":"d32ab786-7fe5-4249-97b1-b4476fc7d7c0","Type":"ContainerStarted","Data":"0b94f57d4c6f92e2491398f925036f59399d3423aa59b1b5bdab7f864eafdef2"} Jul 03 11:13:45.706123 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.706047 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" event={"ID":"d32ab786-7fe5-4249-97b1-b4476fc7d7c0","Type":"ContainerStarted","Data":"f846a31c41f833a4ff510793b7a26e5db76162da741661ddf033ce869aa897c3"} Jul 03 11:13:45.706123 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:45.706070 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" event={"ID":"d32ab786-7fe5-4249-97b1-b4476fc7d7c0","Type":"ContainerStarted","Data":"06fa281bc0328d55af9a682fa6f659d203f98d386d83b27a8c0f47b6085e2f1e"} Jul 03 11:13:46.712270 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:46.712239 2581 generic.go:356] "Generic (PLEG): container finished" podID="61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc" containerID="2c03f9825abd750895fe0507e7fcb2aa0c58ef9f4cd66ee4703bb54c6487de74" exitCode=0 Jul 03 11:13:46.712744 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:46.712302 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkdgv" event={"ID":"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc","Type":"ContainerDied","Data":"2c03f9825abd750895fe0507e7fcb2aa0c58ef9f4cd66ee4703bb54c6487de74"} Jul 03 11:13:47.584841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.584586 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:13:47.718726 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.718687 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkdgv" event={"ID":"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc","Type":"ContainerStarted","Data":"d2e820445e041e95a8d5f8a00f052fa770a0ca17adc4dea3a26c47582980f953"} Jul 03 11:13:47.719144 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.718753 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kkdgv" event={"ID":"61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc","Type":"ContainerStarted","Data":"f57395ca2d794d8ccd862ea933185a30cc3916da4237a493234b367f41b6f4ad"} Jul 03 11:13:47.720778 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.720751 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" event={"ID":"d32ab786-7fe5-4249-97b1-b4476fc7d7c0","Type":"ContainerStarted","Data":"f499e71d30866bddec8a32d7de9f8259fa50ecd7aef1b2b200476ac81fb08d27"} Jul 03 11:13:47.722919 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.722895 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" event={"ID":"c6bcf17a-1e79-4923-b249-6370a485725d","Type":"ContainerStarted","Data":"48c36ba1895bdca17192b951343b5689bca0d7c88390827962faa6d5e5c45c77"} Jul 03 11:13:47.723012 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.722949 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" event={"ID":"c6bcf17a-1e79-4923-b249-6370a485725d","Type":"ContainerStarted","Data":"e04cae06865c90f5634bba808dcaeb573cab20a29c2e1c1c8f3c693339cb1b15"} Jul 03 11:13:47.723012 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.722987 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" event={"ID":"c6bcf17a-1e79-4923-b249-6370a485725d","Type":"ContainerStarted","Data":"1a2d9e2417594ef622fcff56db0d5a4368b849b51edcf4fe4bafe9c38f483fa1"} Jul 03 11:13:47.724338 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.724311 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerID="bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160" exitCode=0 Jul 03 11:13:47.724410 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.724353 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerDied","Data":"bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160"} Jul 03 11:13:47.737027 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.736980 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kkdgv" podStartSLOduration=3.323855258 podStartE2EDuration="4.736967855s" totalImagesPullingTime="1.338579043s" totalInitContainerRuntime="74.533554ms" isStatefulPod=true podCreationTimestamp="2026-07-03 11:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:47.73580878 +0000 UTC m=+90.298484293" watchObservedRunningTime="2026-07-03 11:13:47.736967855 +0000 UTC m=+90.299643372" Jul 03 11:13:47.752782 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.752737 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-54cf96dfcf-pld97" podStartSLOduration=3.329390999 podStartE2EDuration="4.752722823s" totalImagesPullingTime="1.423331824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:47.75165862 +0000 UTC m=+90.314334133" watchObservedRunningTime="2026-07-03 11:13:47.752722823 +0000 UTC m=+90.315398335" Jul 03 11:13:47.794038 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:47.793983 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-7c9dbb8b7d-fd7tm" podStartSLOduration=3.09215457 podStartE2EDuration="4.793966728s" totalImagesPullingTime="1.701812158s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:13:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:47.792697139 +0000 UTC m=+90.355372649" watchObservedRunningTime="2026-07-03 11:13:47.793966728 +0000 UTC m=+90.356642239" Jul 03 11:13:48.478048 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.477984 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-ddcc7f947-mqjf2"] Jul 03 11:13:48.481515 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.481469 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.483789 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.483716 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-49rmj\"" Jul 03 11:13:48.483894 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.483788 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Jul 03 11:13:48.484220 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.484196 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Jul 03 11:13:48.485293 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.484894 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Jul 03 11:13:48.485293 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.484947 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Jul 03 11:13:48.485293 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.484957 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Jul 03 11:13:48.485293 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.485002 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Jul 03 11:13:48.485293 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.484957 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Jul 03 11:13:48.489592 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.489573 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Jul 03 11:13:48.491775 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.491750 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ddcc7f947-mqjf2"] Jul 03 11:13:48.647885 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.647853 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-service-ca\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.648057 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.648001 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-serving-cert\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.648057 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.648050 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cn2kn\" (UniqueName: \"kubernetes.io/projected/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-kube-api-access-cn2kn\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.648150 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.648080 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-oauth-serving-cert\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.648192 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.648162 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-oauth-config\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.648235 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.648196 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-config\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.648235 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.648224 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-trusted-ca-bundle\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.749424 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.749330 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-serving-cert\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.749424 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.749372 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cn2kn\" (UniqueName: \"kubernetes.io/projected/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-kube-api-access-cn2kn\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.749983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.749513 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-oauth-serving-cert\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.749983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.749576 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-oauth-config\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.749983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.749605 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-config\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.749983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.749632 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-trusted-ca-bundle\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.749983 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.749670 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-service-ca\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.750417 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.750361 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-oauth-serving-cert\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.750417 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.750371 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-config\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.751756 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.751248 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-trusted-ca-bundle\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.751756 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.751716 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-service-ca\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.752330 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.752308 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-serving-cert\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.753242 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.753220 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-oauth-config\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.757561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.757543 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cn2kn\" (UniqueName: \"kubernetes.io/projected/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-kube-api-access-cn2kn\") pod \"console-ddcc7f947-mqjf2\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.793846 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.793816 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:48.971840 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:48.971813 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-ddcc7f947-mqjf2"] Jul 03 11:13:48.974771 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:48.974725 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3046ab3_ff2a_4a03_8e94_1ee948b4a76c.slice/crio-2a6a370d3255c2a27380414cd12e5e58fa742f0cf1b0c5e0ef62d99b8bbe7a96 WatchSource:0}: Error finding container 2a6a370d3255c2a27380414cd12e5e58fa742f0cf1b0c5e0ef62d99b8bbe7a96: Status 404 returned error can't find the container with id 2a6a370d3255c2a27380414cd12e5e58fa742f0cf1b0c5e0ef62d99b8bbe7a96 Jul 03 11:13:49.679288 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.679262 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:13:49.684302 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.684282 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.689014 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.688991 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 03 11:13:49.689278 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.689071 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 03 11:13:49.689703 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.689668 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 03 11:13:49.690102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.689854 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 03 11:13:49.690102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.689893 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 03 11:13:49.690102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.689949 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 03 11:13:49.690102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.690017 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 03 11:13:49.690102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.690097 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 03 11:13:49.690322 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.690188 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 03 11:13:49.690322 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.690246 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cfdtg\"" Jul 03 11:13:49.691481 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.690639 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 03 11:13:49.692133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.692111 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 03 11:13:49.692227 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.692190 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bs1506rb0f1vo\"" Jul 03 11:13:49.692347 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.692328 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 03 11:13:49.692421 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.692361 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 03 11:13:49.692635 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.692618 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 03 11:13:49.698609 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.698584 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 03 11:13:49.705480 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.705452 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:13:49.734637 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.734603 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddcc7f947-mqjf2" event={"ID":"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c","Type":"ContainerStarted","Data":"2a6a370d3255c2a27380414cd12e5e58fa742f0cf1b0c5e0ef62d99b8bbe7a96"} Jul 03 11:13:49.737550 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.737525 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerStarted","Data":"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12"} Jul 03 11:13:49.737638 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.737590 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerStarted","Data":"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc"} Jul 03 11:13:49.760176 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760147 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760189 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760237 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760326 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760363 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760414 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760473 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760517 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760547 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760610 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760638 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760663 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760692 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760718 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760742 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760766 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760800 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760836 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760858 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.760896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.760880 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrqsk\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-kube-api-access-wrqsk\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.861590 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.861559 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.861724 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.861605 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.861724 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.861631 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862032 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.861959 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrqsk\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-kube-api-access-wrqsk\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862032 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862028 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862201 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862058 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862201 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862201 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862128 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862201 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862152 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862201 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862185 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862230 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862281 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862303 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862323 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862347 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862393 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862452 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862485 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862510 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.862529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862529 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.863007 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.862544 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.863400 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.863375 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.866479 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.865518 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.866479 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.865612 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.866479 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.865687 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config-out\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.866479 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.866156 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.866734 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.866501 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.867358 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.866934 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.867358 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.866980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.867358 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.867011 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.867865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.867824 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.868529 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.868488 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.868746 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.868727 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.869616 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.869574 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.870241 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.870200 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.870334 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.870286 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.870569 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.870548 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.870789 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.870767 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-web-config\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.871826 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.871802 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrqsk\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-kube-api-access-wrqsk\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.871920 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.871899 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:49.996860 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:49.996825 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:13:50.164488 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.164460 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:13:50.166726 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:13:50.166695 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ba6df6_01ae_42d4_b45c_5b8b36d36f3c.slice/crio-70c55e9ff04ed94f8bbf3528ede4e290e8d7369239ec65222fc11c020e365821 WatchSource:0}: Error finding container 70c55e9ff04ed94f8bbf3528ede4e290e8d7369239ec65222fc11c020e365821: Status 404 returned error can't find the container with id 70c55e9ff04ed94f8bbf3528ede4e290e8d7369239ec65222fc11c020e365821 Jul 03 11:13:50.745662 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.745618 2581 generic.go:356] "Generic (PLEG): container finished" podID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" exitCode=0 Jul 03 11:13:50.745845 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.745727 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerDied","Data":"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d"} Jul 03 11:13:50.745845 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.745824 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerStarted","Data":"70c55e9ff04ed94f8bbf3528ede4e290e8d7369239ec65222fc11c020e365821"} Jul 03 11:13:50.749795 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.749773 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerStarted","Data":"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb"} Jul 03 11:13:50.749896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.749838 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerStarted","Data":"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c"} Jul 03 11:13:50.749896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.749864 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerStarted","Data":"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0"} Jul 03 11:13:50.749896 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.749889 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerStarted","Data":"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878"} Jul 03 11:13:50.802321 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:50.802267 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.342457489 podStartE2EDuration="6.802234285s" totalImagesPullingTime="4.438654592s" totalInitContainerRuntime="21.122204ms" isStatefulPod=false podCreationTimestamp="2026-07-03 11:13:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:50.799559153 +0000 UTC m=+93.362234666" watchObservedRunningTime="2026-07-03 11:13:50.802234285 +0000 UTC m=+93.364909799" Jul 03 11:13:52.760480 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:52.760416 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddcc7f947-mqjf2" event={"ID":"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c","Type":"ContainerStarted","Data":"2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa"} Jul 03 11:13:52.777767 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:52.777708 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-ddcc7f947-mqjf2" podStartSLOduration=1.590507574 podStartE2EDuration="4.777690334s" totalImagesPullingTime="3.18718276s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:13:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:52.777534659 +0000 UTC m=+95.340210172" watchObservedRunningTime="2026-07-03 11:13:52.777690334 +0000 UTC m=+95.340365847" Jul 03 11:13:54.555340 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:54.555302 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846768dbb5-2blkk"] Jul 03 11:13:55.564574 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:55.564546 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-76w8c" Jul 03 11:13:55.776360 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:55.776328 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerStarted","Data":"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc"} Jul 03 11:13:55.776554 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:55.776392 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerStarted","Data":"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4"} Jul 03 11:13:57.788680 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:57.788646 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerStarted","Data":"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58"} Jul 03 11:13:57.789028 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:57.788738 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerStarted","Data":"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c"} Jul 03 11:13:57.789028 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:57.788769 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerStarted","Data":"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7"} Jul 03 11:13:57.789028 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:57.788793 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerStarted","Data":"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110"} Jul 03 11:13:57.817425 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:57.817371 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.378968875 podStartE2EDuration="8.817353184s" totalImagesPullingTime="6.418358075s" totalInitContainerRuntime="20.026234ms" isStatefulPod=false podCreationTimestamp="2026-07-03 11:13:49 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:13:57.814469103 +0000 UTC m=+100.377144614" watchObservedRunningTime="2026-07-03 11:13:57.817353184 +0000 UTC m=+100.380028711" Jul 03 11:13:58.794347 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:58.794318 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:58.794854 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:58.794373 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:58.799107 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:58.799082 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:59.800753 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:59.800724 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:13:59.997585 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:13:59.997557 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:14:14.861197 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:14.861162 2581 generic.go:356] "Generic (PLEG): container finished" podID="c1477b24-cb02-4165-925e-8146aa836065" containerID="a465ec77c6ddf7a3f00c32d16d861b9824e8cd6cac3f6f2f6d56adbdec1bb576" exitCode=0 Jul 03 11:14:14.861678 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:14.861241 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" event={"ID":"c1477b24-cb02-4165-925e-8146aa836065","Type":"ContainerDied","Data":"a465ec77c6ddf7a3f00c32d16d861b9824e8cd6cac3f6f2f6d56adbdec1bb576"} Jul 03 11:14:14.861743 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:14.861690 2581 scope.go:117] "RemoveContainer" containerID="a465ec77c6ddf7a3f00c32d16d861b9824e8cd6cac3f6f2f6d56adbdec1bb576" Jul 03 11:14:15.867113 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:15.867080 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-846bf668f7-ln9mh" event={"ID":"c1477b24-cb02-4165-925e-8146aa836065","Type":"ContainerStarted","Data":"d3cc479af0cf55fe43428010b5277d123e756bffac49a839b9a05f870deb2030"} Jul 03 11:14:19.581247 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.581178 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" podUID="db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" containerName="registry" containerID="cri-o://545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743" gracePeriod=30 Jul 03 11:14:19.826490 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.826464 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:14:19.886720 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.886682 2581 generic.go:356] "Generic (PLEG): container finished" podID="9885b117-aabd-414f-be44-5ee5f7c908eb" containerID="501c154df6871852f186685ff020897f78a156da54e583f799b822dcac588eac" exitCode=0 Jul 03 11:14:19.886900 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.886750 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" event={"ID":"9885b117-aabd-414f-be44-5ee5f7c908eb","Type":"ContainerDied","Data":"501c154df6871852f186685ff020897f78a156da54e583f799b822dcac588eac"} Jul 03 11:14:19.887191 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.887168 2581 scope.go:117] "RemoveContainer" containerID="501c154df6871852f186685ff020897f78a156da54e583f799b822dcac588eac" Jul 03 11:14:19.888221 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.888193 2581 generic.go:356] "Generic (PLEG): container finished" podID="db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" containerID="545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743" exitCode=0 Jul 03 11:14:19.888332 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.888256 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" Jul 03 11:14:19.888332 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.888282 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" event={"ID":"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11","Type":"ContainerDied","Data":"545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743"} Jul 03 11:14:19.888452 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.888338 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-846768dbb5-2blkk" event={"ID":"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11","Type":"ContainerDied","Data":"c343e34cfad090445f3e76fa47bfee9d37170bc2e6113f8f5c173cd75c72475e"} Jul 03 11:14:19.888452 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.888365 2581 scope.go:117] "RemoveContainer" containerID="545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743" Jul 03 11:14:19.898332 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.898311 2581 scope.go:117] "RemoveContainer" containerID="545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743" Jul 03 11:14:19.898704 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:14:19.898676 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743\": container with ID starting with 545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743 not found: ID does not exist" containerID="545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743" Jul 03 11:14:19.898786 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.898716 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743"} err="failed to get container status \"545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743\": rpc error: code = NotFound desc = could not find container \"545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743\": container with ID starting with 545ad779c74fb8132a86f008f62f5e3176594774869968f2b3ed92c08225e743 not found: ID does not exist" Jul 03 11:14:19.956014 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.955982 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") pod \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " Jul 03 11:14:19.956168 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.956071 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-image-registry-private-configuration\") pod \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " Jul 03 11:14:19.956168 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.956099 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-certificates\") pod \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " Jul 03 11:14:19.956168 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.956116 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lczsw\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-kube-api-access-lczsw\") pod \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " Jul 03 11:14:19.956168 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.956144 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-bound-sa-token\") pod \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " Jul 03 11:14:19.956379 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.956205 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-installation-pull-secrets\") pod \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " Jul 03 11:14:19.956379 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.956251 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-trusted-ca\") pod \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " Jul 03 11:14:19.956379 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.956288 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-ca-trust-extracted\") pod \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\" (UID: \"db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11\") " Jul 03 11:14:19.957159 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.956911 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:14:19.958221 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.958185 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:14:19.959187 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.959161 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-kube-api-access-lczsw" (OuterVolumeSpecName: "kube-api-access-lczsw") pod "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11"). InnerVolumeSpecName "kube-api-access-lczsw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:14:19.959317 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.959262 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:14:19.960093 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.960066 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:14:19.960223 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.960195 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:14:19.960390 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.960369 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:14:19.968188 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:19.968153 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" (UID: "db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:14:20.057319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.057280 2581 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-tls\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:14:20.057319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.057312 2581 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-image-registry-private-configuration\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:14:20.057319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.057322 2581 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-registry-certificates\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:14:20.057319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.057331 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lczsw\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-kube-api-access-lczsw\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:14:20.057626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.057340 2581 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-bound-sa-token\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:14:20.057626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.057348 2581 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-installation-pull-secrets\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:14:20.057626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.057358 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-trusted-ca\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:14:20.057626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.057366 2581 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11-ca-trust-extracted\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:14:20.204318 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.204231 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-846768dbb5-2blkk"] Jul 03 11:14:20.208262 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.208225 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-846768dbb5-2blkk"] Jul 03 11:14:20.895375 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:20.895335 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d46d9ffd4-vhznm" event={"ID":"9885b117-aabd-414f-be44-5ee5f7c908eb","Type":"ContainerStarted","Data":"bcaeab7a5f32003f3746dff1e01f4a71c03e9042b186dfc21289b26d3c012bfc"} Jul 03 11:14:21.994578 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:21.994544 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" path="/var/lib/kubelet/pods/db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11/volumes" Jul 03 11:14:28.930362 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:28.930312 2581 generic.go:356] "Generic (PLEG): container finished" podID="4b09b25f-d483-4e17-8c66-76fc0dbc1bf7" containerID="b1dde5dda9ff06fb32280f621897d8cc30d543b347046203a7e786cccd72b90d" exitCode=0 Jul 03 11:14:28.930769 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:28.930387 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" event={"ID":"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7","Type":"ContainerDied","Data":"b1dde5dda9ff06fb32280f621897d8cc30d543b347046203a7e786cccd72b90d"} Jul 03 11:14:28.930823 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:28.930772 2581 scope.go:117] "RemoveContainer" containerID="b1dde5dda9ff06fb32280f621897d8cc30d543b347046203a7e786cccd72b90d" Jul 03 11:14:29.509665 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:29.509634 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678cfbb59c-mm6jx_6ef1ae29-90df-42ac-8b5d-049e00ffe01d/router/0.log" Jul 03 11:14:29.523558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:29.523535 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rzn28_c5c70232-5339-4786-bf98-86283e335ea1/serve-healthcheck-canary/0.log" Jul 03 11:14:29.936165 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:29.936134 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-5c847dcfc4-7wpk5" event={"ID":"4b09b25f-d483-4e17-8c66-76fc0dbc1bf7","Type":"ContainerStarted","Data":"f65223c7ee48674a9d4fd3245bacf70907a7e407a22bc5bd49e565c4fc9907dd"} Jul 03 11:14:49.997791 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:49.997758 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:14:50.014369 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:50.014338 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:14:50.049256 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:14:50.049226 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:03.719649 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:03.719611 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:15:03.720067 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:03.719876 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="alertmanager" containerID="cri-o://89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc" gracePeriod=120 Jul 03 11:15:03.720067 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:03.719939 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy-metric" containerID="cri-o://3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c" gracePeriod=120 Jul 03 11:15:03.720067 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:03.719959 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy-web" containerID="cri-o://2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878" gracePeriod=120 Jul 03 11:15:03.720067 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:03.720015 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="prom-label-proxy" containerID="cri-o://002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb" gracePeriod=120 Jul 03 11:15:03.720067 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:03.720041 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="config-reloader" containerID="cri-o://26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12" gracePeriod=120 Jul 03 11:15:03.720322 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:03.720024 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy" containerID="cri-o://52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0" gracePeriod=120 Jul 03 11:15:04.096427 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.096347 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerID="002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb" exitCode=0 Jul 03 11:15:04.096427 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.096374 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerID="52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0" exitCode=0 Jul 03 11:15:04.096427 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.096384 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerID="26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12" exitCode=0 Jul 03 11:15:04.096427 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.096393 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerID="89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc" exitCode=0 Jul 03 11:15:04.096427 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.096415 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerDied","Data":"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb"} Jul 03 11:15:04.096719 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.096505 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerDied","Data":"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0"} Jul 03 11:15:04.096719 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.096532 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerDied","Data":"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12"} Jul 03 11:15:04.096719 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.096557 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerDied","Data":"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc"} Jul 03 11:15:04.968299 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:04.968275 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.082770 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082685 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-config-out\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.082770 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082735 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-main-tls\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.082770 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082766 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083044 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082793 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-trusted-ca-bundle\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083044 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082811 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6tks7\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-kube-api-access-6tks7\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083044 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082837 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-metrics-client-ca\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083044 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082870 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-main-db\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083044 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082898 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-tls-assets\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083044 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082925 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-config-volume\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083044 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082971 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-web-config\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083044 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.082994 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-metric\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083426 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.083051 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-cluster-tls-config\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.083426 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.083100 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-web\") pod \"b3d062e1-3c80-434c-bd10-df89727e48fb\" (UID: \"b3d062e1-3c80-434c-bd10-df89727e48fb\") " Jul 03 11:15:05.084094 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.083457 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:05.084205 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.083610 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:15:05.084205 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.083846 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:05.086420 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.086353 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-config-out" (OuterVolumeSpecName: "config-out") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:15:05.086420 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.086372 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:05.086420 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.086409 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:05.086647 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.086494 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:15:05.086809 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.086774 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:05.086888 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.086866 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-config-volume" (OuterVolumeSpecName: "config-volume") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:05.087067 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.087048 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:05.087571 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.087553 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-kube-api-access-6tks7" (OuterVolumeSpecName: "kube-api-access-6tks7") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "kube-api-access-6tks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:15:05.091066 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.091037 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:05.099219 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.099187 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-web-config" (OuterVolumeSpecName: "web-config") pod "b3d062e1-3c80-434c-bd10-df89727e48fb" (UID: "b3d062e1-3c80-434c-bd10-df89727e48fb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:05.104581 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.104554 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerID="3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c" exitCode=0 Jul 03 11:15:05.104581 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.104580 2581 generic.go:356] "Generic (PLEG): container finished" podID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerID="2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878" exitCode=0 Jul 03 11:15:05.104756 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.104613 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerDied","Data":"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c"} Jul 03 11:15:05.104756 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.104632 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.104756 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.104691 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerDied","Data":"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878"} Jul 03 11:15:05.104756 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.104737 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"b3d062e1-3c80-434c-bd10-df89727e48fb","Type":"ContainerDied","Data":"8441f2afe6c66ebf347dacaf7688bd78928ec5f8c404f0f6806ec5db17ce6724"} Jul 03 11:15:05.104903 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.104757 2581 scope.go:117] "RemoveContainer" containerID="002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb" Jul 03 11:15:05.113666 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.113650 2581 scope.go:117] "RemoveContainer" containerID="3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c" Jul 03 11:15:05.121148 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.121132 2581 scope.go:117] "RemoveContainer" containerID="52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0" Jul 03 11:15:05.128927 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.128893 2581 scope.go:117] "RemoveContainer" containerID="2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878" Jul 03 11:15:05.130701 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.130682 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:15:05.135302 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.135277 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:15:05.137252 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.137232 2581 scope.go:117] "RemoveContainer" containerID="26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12" Jul 03 11:15:05.144504 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.144480 2581 scope.go:117] "RemoveContainer" containerID="89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc" Jul 03 11:15:05.152135 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.152111 2581 scope.go:117] "RemoveContainer" containerID="bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160" Jul 03 11:15:05.160388 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.160363 2581 scope.go:117] "RemoveContainer" containerID="002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb" Jul 03 11:15:05.161142 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:05.161101 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb\": container with ID starting with 002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb not found: ID does not exist" containerID="002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb" Jul 03 11:15:05.161267 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.161154 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb"} err="failed to get container status \"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb\": rpc error: code = NotFound desc = could not find container \"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb\": container with ID starting with 002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb not found: ID does not exist" Jul 03 11:15:05.161267 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.161179 2581 scope.go:117] "RemoveContainer" containerID="3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c" Jul 03 11:15:05.161541 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:05.161517 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c\": container with ID starting with 3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c not found: ID does not exist" containerID="3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c" Jul 03 11:15:05.161609 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.161550 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c"} err="failed to get container status \"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c\": rpc error: code = NotFound desc = could not find container \"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c\": container with ID starting with 3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c not found: ID does not exist" Jul 03 11:15:05.161609 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.161573 2581 scope.go:117] "RemoveContainer" containerID="52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0" Jul 03 11:15:05.161832 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:05.161815 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0\": container with ID starting with 52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0 not found: ID does not exist" containerID="52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0" Jul 03 11:15:05.161885 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.161843 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0"} err="failed to get container status \"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0\": rpc error: code = NotFound desc = could not find container \"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0\": container with ID starting with 52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0 not found: ID does not exist" Jul 03 11:15:05.161885 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.161860 2581 scope.go:117] "RemoveContainer" containerID="2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878" Jul 03 11:15:05.162097 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:05.162082 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878\": container with ID starting with 2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878 not found: ID does not exist" containerID="2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878" Jul 03 11:15:05.162142 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162102 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878"} err="failed to get container status \"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878\": rpc error: code = NotFound desc = could not find container \"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878\": container with ID starting with 2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878 not found: ID does not exist" Jul 03 11:15:05.162142 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162124 2581 scope.go:117] "RemoveContainer" containerID="26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12" Jul 03 11:15:05.162375 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:05.162357 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12\": container with ID starting with 26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12 not found: ID does not exist" containerID="26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12" Jul 03 11:15:05.162536 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162381 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12"} err="failed to get container status \"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12\": rpc error: code = NotFound desc = could not find container \"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12\": container with ID starting with 26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12 not found: ID does not exist" Jul 03 11:15:05.162536 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162402 2581 scope.go:117] "RemoveContainer" containerID="89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc" Jul 03 11:15:05.162629 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162562 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:15:05.162664 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:05.162631 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc\": container with ID starting with 89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc not found: ID does not exist" containerID="89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc" Jul 03 11:15:05.162664 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162647 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc"} err="failed to get container status \"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc\": rpc error: code = NotFound desc = could not find container \"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc\": container with ID starting with 89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc not found: ID does not exist" Jul 03 11:15:05.162664 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162661 2581 scope.go:117] "RemoveContainer" containerID="bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160" Jul 03 11:15:05.162911 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:05.162897 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160\": container with ID starting with bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160 not found: ID does not exist" containerID="bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160" Jul 03 11:15:05.162957 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162914 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160"} err="failed to get container status \"bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160\": rpc error: code = NotFound desc = could not find container \"bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160\": container with ID starting with bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160 not found: ID does not exist" Jul 03 11:15:05.162957 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162926 2581 scope.go:117] "RemoveContainer" containerID="002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb" Jul 03 11:15:05.163035 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162965 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" containerName="registry" Jul 03 11:15:05.163035 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162979 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" containerName="registry" Jul 03 11:15:05.163035 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162990 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy-web" Jul 03 11:15:05.163035 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.162996 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy-web" Jul 03 11:15:05.163035 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163013 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="init-config-reloader" Jul 03 11:15:05.163035 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163024 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="init-config-reloader" Jul 03 11:15:05.163035 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163033 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="config-reloader" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163041 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="config-reloader" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163049 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163057 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163066 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy-metric" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163072 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy-metric" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163084 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="prom-label-proxy" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163089 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="prom-label-proxy" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163095 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="alertmanager" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163100 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="alertmanager" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163107 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb"} err="failed to get container status \"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb\": rpc error: code = NotFound desc = could not find container \"002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb\": container with ID starting with 002d5bd9bb85a70d4689ec1516020ce7c7e31c541b2bbdfde5459861b9d9bdfb not found: ID does not exist" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163121 2581 scope.go:117] "RemoveContainer" containerID="3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163159 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="alertmanager" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163167 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy-metric" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163175 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="prom-label-proxy" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163181 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="db37c72c-8cdf-4aff-a2dc-59d6a9f4eb11" containerName="registry" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163187 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="config-reloader" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163192 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy-web" Jul 03 11:15:05.163255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163197 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" containerName="kube-rbac-proxy" Jul 03 11:15:05.164021 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163336 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c"} err="failed to get container status \"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c\": rpc error: code = NotFound desc = could not find container \"3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c\": container with ID starting with 3218cfcf8ecdcff128d9526f4f10ba8e13ec29c963d5e9b6b55cb0cfbcbeb74c not found: ID does not exist" Jul 03 11:15:05.164021 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163362 2581 scope.go:117] "RemoveContainer" containerID="52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0" Jul 03 11:15:05.164021 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163602 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0"} err="failed to get container status \"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0\": rpc error: code = NotFound desc = could not find container \"52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0\": container with ID starting with 52a42cff6fcd8201bcb23b696ecfc05985019050342ffe77bd530ed80b5a96d0 not found: ID does not exist" Jul 03 11:15:05.164021 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163619 2581 scope.go:117] "RemoveContainer" containerID="2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878" Jul 03 11:15:05.164021 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163858 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878"} err="failed to get container status \"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878\": rpc error: code = NotFound desc = could not find container \"2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878\": container with ID starting with 2c0b2a0a257a0c9bc0dc2f3ddd2dab64e1760bfcc4a42f14385b1ac6a26e1878 not found: ID does not exist" Jul 03 11:15:05.164021 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.163873 2581 scope.go:117] "RemoveContainer" containerID="26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12" Jul 03 11:15:05.164272 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.164076 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12"} err="failed to get container status \"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12\": rpc error: code = NotFound desc = could not find container \"26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12\": container with ID starting with 26e75e0f97d95d2b86bd88a423302daee7de34ec298cced1f1d09e9f5bcbee12 not found: ID does not exist" Jul 03 11:15:05.164272 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.164090 2581 scope.go:117] "RemoveContainer" containerID="89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc" Jul 03 11:15:05.164346 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.164294 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc"} err="failed to get container status \"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc\": rpc error: code = NotFound desc = could not find container \"89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc\": container with ID starting with 89b2995ef3ec00fb79f2327559357d9bb42f21490141307c76efdfa5c83c35dc not found: ID does not exist" Jul 03 11:15:05.164346 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.164317 2581 scope.go:117] "RemoveContainer" containerID="bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160" Jul 03 11:15:05.164627 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.164607 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160"} err="failed to get container status \"bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160\": rpc error: code = NotFound desc = could not find container \"bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160\": container with ID starting with bbaa8b6dd385e39cd6e09a2c47d32d5bb8021414d425acd55bfee0b88bc8c160 not found: ID does not exist" Jul 03 11:15:05.166971 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.166955 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.169671 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.169653 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Jul 03 11:15:05.169771 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.169659 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Jul 03 11:15:05.169941 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.169926 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Jul 03 11:15:05.170214 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.170199 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Jul 03 11:15:05.170295 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.170229 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Jul 03 11:15:05.170445 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.170406 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Jul 03 11:15:05.170445 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.170423 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-pbnqg\"" Jul 03 11:15:05.170605 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.170461 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Jul 03 11:15:05.170605 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.170423 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Jul 03 11:15:05.180857 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.177779 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Jul 03 11:15:05.183168 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.183140 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:15:05.184784 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184767 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-config-out\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184787 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-main-tls\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184798 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184808 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184818 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tks7\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-kube-api-access-6tks7\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184826 2581 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b3d062e1-3c80-434c-bd10-df89727e48fb-metrics-client-ca\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184835 2581 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/b3d062e1-3c80-434c-bd10-df89727e48fb-alertmanager-main-db\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184842 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b3d062e1-3c80-434c-bd10-df89727e48fb-tls-assets\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184851 2581 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-config-volume\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184858 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-web-config\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.184865 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184866 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.185190 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184875 2581 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-cluster-tls-config\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.185190 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.184887 2581 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b3d062e1-3c80-434c-bd10-df89727e48fb-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:05.285774 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.285728 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ecf47df-010c-4dea-8449-b96df2e4cbcd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.285774 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.285783 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286043 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.285844 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286043 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.285894 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5qml\" (UniqueName: \"kubernetes.io/projected/3ecf47df-010c-4dea-8449-b96df2e4cbcd-kube-api-access-x5qml\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286043 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.285932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286043 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.286004 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286043 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.286031 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-web-config\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286290 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.286070 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ecf47df-010c-4dea-8449-b96df2e4cbcd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286290 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.286104 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ecf47df-010c-4dea-8449-b96df2e4cbcd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286290 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.286135 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286290 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.286158 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ecf47df-010c-4dea-8449-b96df2e4cbcd-config-out\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286290 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.286190 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecf47df-010c-4dea-8449-b96df2e4cbcd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.286290 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.286223 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.386895 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.386863 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-web-config\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387039 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.386906 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ecf47df-010c-4dea-8449-b96df2e4cbcd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387039 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.386929 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ecf47df-010c-4dea-8449-b96df2e4cbcd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387039 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.386957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387039 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.386992 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ecf47df-010c-4dea-8449-b96df2e4cbcd-config-out\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387039 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387019 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecf47df-010c-4dea-8449-b96df2e4cbcd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387317 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387050 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387317 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387076 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ecf47df-010c-4dea-8449-b96df2e4cbcd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387317 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387113 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387317 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387140 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387317 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387162 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5qml\" (UniqueName: \"kubernetes.io/projected/3ecf47df-010c-4dea-8449-b96df2e4cbcd-kube-api-access-x5qml\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387317 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387195 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387317 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.387683 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387339 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/3ecf47df-010c-4dea-8449-b96df2e4cbcd-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.388424 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.387785 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3ecf47df-010c-4dea-8449-b96df2e4cbcd-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.388424 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.388207 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3ecf47df-010c-4dea-8449-b96df2e4cbcd-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.390189 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.390044 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3ecf47df-010c-4dea-8449-b96df2e4cbcd-config-out\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.390400 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.390332 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.390534 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.390397 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-web-config\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.390534 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.390397 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-config-volume\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.390534 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.390482 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.390650 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.390585 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.390951 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.390934 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.390992 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.390978 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3ecf47df-010c-4dea-8449-b96df2e4cbcd-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.392150 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.392130 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3ecf47df-010c-4dea-8449-b96df2e4cbcd-tls-assets\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.396211 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.396191 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5qml\" (UniqueName: \"kubernetes.io/projected/3ecf47df-010c-4dea-8449-b96df2e4cbcd-kube-api-access-x5qml\") pod \"alertmanager-main-0\" (UID: \"3ecf47df-010c-4dea-8449-b96df2e4cbcd\") " pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.483802 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.483765 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Jul 03 11:15:05.629822 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.629778 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Jul 03 11:15:05.632606 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:15:05.632575 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ecf47df_010c_4dea_8449_b96df2e4cbcd.slice/crio-2835517584e64e39b35954e24c88d15ddb6483891c381487558735cfe7ec9125 WatchSource:0}: Error finding container 2835517584e64e39b35954e24c88d15ddb6483891c381487558735cfe7ec9125: Status 404 returned error can't find the container with id 2835517584e64e39b35954e24c88d15ddb6483891c381487558735cfe7ec9125 Jul 03 11:15:05.993622 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:05.993543 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3d062e1-3c80-434c-bd10-df89727e48fb" path="/var/lib/kubelet/pods/b3d062e1-3c80-434c-bd10-df89727e48fb/volumes" Jul 03 11:15:06.111520 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:06.111479 2581 generic.go:356] "Generic (PLEG): container finished" podID="3ecf47df-010c-4dea-8449-b96df2e4cbcd" containerID="8ff5bb2a1b284ad47dbfaf71ea4208e2c7221c86fc1902dc121d926a2b200c32" exitCode=0 Jul 03 11:15:06.111683 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:06.111542 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecf47df-010c-4dea-8449-b96df2e4cbcd","Type":"ContainerDied","Data":"8ff5bb2a1b284ad47dbfaf71ea4208e2c7221c86fc1902dc121d926a2b200c32"} Jul 03 11:15:06.111683 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:06.111606 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecf47df-010c-4dea-8449-b96df2e4cbcd","Type":"ContainerStarted","Data":"2835517584e64e39b35954e24c88d15ddb6483891c381487558735cfe7ec9125"} Jul 03 11:15:07.118934 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:07.118898 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecf47df-010c-4dea-8449-b96df2e4cbcd","Type":"ContainerStarted","Data":"20f0531b2b286fbb1631005c8b343af5a5f36f798c1b79237f01d578e8314431"} Jul 03 11:15:07.119611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:07.118961 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecf47df-010c-4dea-8449-b96df2e4cbcd","Type":"ContainerStarted","Data":"ce88c46d42e21b2ebff0f7f4017cdad2a55d27631d4ea65468b94dc02003fbc1"} Jul 03 11:15:07.119611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:07.118990 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecf47df-010c-4dea-8449-b96df2e4cbcd","Type":"ContainerStarted","Data":"19f75300dd48814f871ec67bb30dcb23eb645f092319430981bc2a3a03c9fb79"} Jul 03 11:15:07.119611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:07.119016 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecf47df-010c-4dea-8449-b96df2e4cbcd","Type":"ContainerStarted","Data":"a806365fb34b331e710d627163513a87f76a2fae176317b071e944e17f05aee2"} Jul 03 11:15:07.119611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:07.119044 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecf47df-010c-4dea-8449-b96df2e4cbcd","Type":"ContainerStarted","Data":"75e55b4d35ee1cce17940b47decfe04124f72af1f8b48e16ec0566446ca3bdf5"} Jul 03 11:15:07.119611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:07.119068 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"3ecf47df-010c-4dea-8449-b96df2e4cbcd","Type":"ContainerStarted","Data":"c54237581df3162c1e9ba9cc83173efc626110bf59e1947a77bf871952c278cf"} Jul 03 11:15:07.152032 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:07.151977 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.136674517 podStartE2EDuration="2.151958924s" totalImagesPullingTime="0s" totalInitContainerRuntime="15.284407ms" isStatefulPod=false podCreationTimestamp="2026-07-03 11:15:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:15:07.149980018 +0000 UTC m=+169.712655531" watchObservedRunningTime="2026-07-03 11:15:07.151958924 +0000 UTC m=+169.714634436" Jul 03 11:15:08.099854 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.099767 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:15:08.100153 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.100117 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="prometheus" containerID="cri-o://30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" gracePeriod=600 Jul 03 11:15:08.100254 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.100129 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" gracePeriod=600 Jul 03 11:15:08.100254 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.100156 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy-web" containerID="cri-o://a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" gracePeriod=600 Jul 03 11:15:08.100254 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.100174 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="config-reloader" containerID="cri-o://038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" gracePeriod=600 Jul 03 11:15:08.100254 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.100124 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="thanos-sidecar" containerID="cri-o://d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" gracePeriod=600 Jul 03 11:15:08.100254 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.100181 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy" containerID="cri-o://a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" gracePeriod=600 Jul 03 11:15:08.364556 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.364531 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:08.517386 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517343 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-serving-certs-ca-bundle\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517386 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517384 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-thanos-prometheus-http-client-file\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517404 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517426 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-trusted-ca-bundle\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517483 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-web-config\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517517 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-db\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517560 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config-out\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517626 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517614 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-2\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517635 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-metrics-client-certs\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517659 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-kubelet-serving-ca-bundle\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517705 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517746 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517774 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-0\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517812 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-grpc-tls\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517836 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-tls\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517831 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517863 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-tls-assets\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517889 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wrqsk\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-kube-api-access-wrqsk\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517918 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:08.517942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517922 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-kube-rbac-proxy\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.518738 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.517971 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-1\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.518738 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.518107 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-metrics-client-ca\") pod \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\" (UID: \"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c\") " Jul 03 11:15:08.518738 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.518132 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:08.518738 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.518408 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.518738 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.518494 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.518738 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.518630 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.519031 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.518884 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:08.519031 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.518972 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:08.519226 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.519114 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:15:08.519350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.519326 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:08.519727 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.519701 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:08.520270 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.520219 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.520525 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.520484 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.520781 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.520584 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.522230 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.522202 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config-out" (OuterVolumeSpecName: "config-out") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:15:08.522417 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.522394 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.522598 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.522575 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config" (OuterVolumeSpecName: "config") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.522677 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.522649 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.522677 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.522659 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-kube-api-access-wrqsk" (OuterVolumeSpecName: "kube-api-access-wrqsk") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "kube-api-access-wrqsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:15:08.522839 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.522805 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.522965 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.522946 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.523233 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.523214 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:15:08.533842 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.533813 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-web-config" (OuterVolumeSpecName: "web-config") pod "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" (UID: "60ba6df6-01ae-42d4-b45c-5b8b36d36f3c"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:08.619454 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619348 2581 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config-out\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619454 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619379 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619454 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619389 2581 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-metrics-client-certs\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619454 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619400 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619454 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619410 2581 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-config\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619454 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619419 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619454 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619451 2581 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-grpc-tls\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619454 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619461 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619469 2581 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-tls-assets\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619478 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wrqsk\" (UniqueName: \"kubernetes.io/projected/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-kube-api-access-wrqsk\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619486 2581 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-kube-rbac-proxy\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619494 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619503 2581 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-configmap-metrics-client-ca\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619511 2581 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619522 2581 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619531 2581 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-web-config\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:08.619799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:08.619538 2581 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c-prometheus-k8s-db\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:09.132921 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.132886 2581 generic.go:356] "Generic (PLEG): container finished" podID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" exitCode=0 Jul 03 11:15:09.132921 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.132915 2581 generic.go:356] "Generic (PLEG): container finished" podID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" exitCode=0 Jul 03 11:15:09.132921 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.132922 2581 generic.go:356] "Generic (PLEG): container finished" podID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" exitCode=0 Jul 03 11:15:09.132921 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.132928 2581 generic.go:356] "Generic (PLEG): container finished" podID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" exitCode=0 Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.132935 2581 generic.go:356] "Generic (PLEG): container finished" podID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" exitCode=0 Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.132941 2581 generic.go:356] "Generic (PLEG): container finished" podID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" exitCode=0 Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.132961 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.132973 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerDied","Data":"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58"} Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.133054 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerDied","Data":"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c"} Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.133084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerDied","Data":"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7"} Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.133111 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerDied","Data":"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110"} Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.133123 2581 scope.go:117] "RemoveContainer" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" Jul 03 11:15:09.133157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.133140 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerDied","Data":"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc"} Jul 03 11:15:09.133489 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.133165 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerDied","Data":"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4"} Jul 03 11:15:09.133489 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.133191 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"60ba6df6-01ae-42d4-b45c-5b8b36d36f3c","Type":"ContainerDied","Data":"70c55e9ff04ed94f8bbf3528ede4e290e8d7369239ec65222fc11c020e365821"} Jul 03 11:15:09.141704 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.141686 2581 scope.go:117] "RemoveContainer" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" Jul 03 11:15:09.149028 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.149010 2581 scope.go:117] "RemoveContainer" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" Jul 03 11:15:09.156227 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.156205 2581 scope.go:117] "RemoveContainer" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" Jul 03 11:15:09.159507 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.159486 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:15:09.164140 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.164122 2581 scope.go:117] "RemoveContainer" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" Jul 03 11:15:09.168973 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.168949 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:15:09.172428 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.172404 2581 scope.go:117] "RemoveContainer" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" Jul 03 11:15:09.180765 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.180744 2581 scope.go:117] "RemoveContainer" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" Jul 03 11:15:09.188318 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.188297 2581 scope.go:117] "RemoveContainer" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" Jul 03 11:15:09.188615 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:09.188595 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": container with ID starting with 3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58 not found: ID does not exist" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" Jul 03 11:15:09.188682 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.188625 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58"} err="failed to get container status \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": rpc error: code = NotFound desc = could not find container \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": container with ID starting with 3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58 not found: ID does not exist" Jul 03 11:15:09.188682 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.188645 2581 scope.go:117] "RemoveContainer" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" Jul 03 11:15:09.188885 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:09.188867 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": container with ID starting with a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c not found: ID does not exist" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" Jul 03 11:15:09.188930 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.188889 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c"} err="failed to get container status \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": rpc error: code = NotFound desc = could not find container \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": container with ID starting with a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c not found: ID does not exist" Jul 03 11:15:09.188930 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.188902 2581 scope.go:117] "RemoveContainer" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" Jul 03 11:15:09.189131 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:09.189115 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": container with ID starting with a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7 not found: ID does not exist" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" Jul 03 11:15:09.189175 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.189134 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7"} err="failed to get container status \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": rpc error: code = NotFound desc = could not find container \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": container with ID starting with a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7 not found: ID does not exist" Jul 03 11:15:09.189175 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.189149 2581 scope.go:117] "RemoveContainer" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" Jul 03 11:15:09.189374 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:09.189356 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": container with ID starting with d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110 not found: ID does not exist" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" Jul 03 11:15:09.189453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.189383 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110"} err="failed to get container status \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": rpc error: code = NotFound desc = could not find container \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": container with ID starting with d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110 not found: ID does not exist" Jul 03 11:15:09.189453 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.189406 2581 scope.go:117] "RemoveContainer" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" Jul 03 11:15:09.189674 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:09.189658 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": container with ID starting with 038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc not found: ID does not exist" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" Jul 03 11:15:09.189709 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.189679 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc"} err="failed to get container status \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": rpc error: code = NotFound desc = could not find container \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": container with ID starting with 038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc not found: ID does not exist" Jul 03 11:15:09.189709 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.189693 2581 scope.go:117] "RemoveContainer" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" Jul 03 11:15:09.189917 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:09.189903 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": container with ID starting with 30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4 not found: ID does not exist" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" Jul 03 11:15:09.189972 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.189921 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4"} err="failed to get container status \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": rpc error: code = NotFound desc = could not find container \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": container with ID starting with 30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4 not found: ID does not exist" Jul 03 11:15:09.189972 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.189932 2581 scope.go:117] "RemoveContainer" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" Jul 03 11:15:09.190166 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:09.190149 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": container with ID starting with ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d not found: ID does not exist" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" Jul 03 11:15:09.190198 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.190173 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d"} err="failed to get container status \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": rpc error: code = NotFound desc = could not find container \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": container with ID starting with ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d not found: ID does not exist" Jul 03 11:15:09.190198 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.190188 2581 scope.go:117] "RemoveContainer" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" Jul 03 11:15:09.190416 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.190398 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58"} err="failed to get container status \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": rpc error: code = NotFound desc = could not find container \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": container with ID starting with 3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58 not found: ID does not exist" Jul 03 11:15:09.190513 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.190417 2581 scope.go:117] "RemoveContainer" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" Jul 03 11:15:09.190674 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.190655 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c"} err="failed to get container status \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": rpc error: code = NotFound desc = could not find container \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": container with ID starting with a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c not found: ID does not exist" Jul 03 11:15:09.190717 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.190676 2581 scope.go:117] "RemoveContainer" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" Jul 03 11:15:09.190900 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.190883 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7"} err="failed to get container status \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": rpc error: code = NotFound desc = could not find container \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": container with ID starting with a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7 not found: ID does not exist" Jul 03 11:15:09.190900 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.190902 2581 scope.go:117] "RemoveContainer" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" Jul 03 11:15:09.191126 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191108 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110"} err="failed to get container status \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": rpc error: code = NotFound desc = could not find container \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": container with ID starting with d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110 not found: ID does not exist" Jul 03 11:15:09.191176 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191127 2581 scope.go:117] "RemoveContainer" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" Jul 03 11:15:09.191332 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191318 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc"} err="failed to get container status \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": rpc error: code = NotFound desc = could not find container \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": container with ID starting with 038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc not found: ID does not exist" Jul 03 11:15:09.191371 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191332 2581 scope.go:117] "RemoveContainer" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" Jul 03 11:15:09.191580 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191561 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4"} err="failed to get container status \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": rpc error: code = NotFound desc = could not find container \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": container with ID starting with 30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4 not found: ID does not exist" Jul 03 11:15:09.191635 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191583 2581 scope.go:117] "RemoveContainer" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" Jul 03 11:15:09.191772 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191752 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d"} err="failed to get container status \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": rpc error: code = NotFound desc = could not find container \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": container with ID starting with ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d not found: ID does not exist" Jul 03 11:15:09.191772 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191768 2581 scope.go:117] "RemoveContainer" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" Jul 03 11:15:09.192007 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.191984 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58"} err="failed to get container status \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": rpc error: code = NotFound desc = could not find container \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": container with ID starting with 3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58 not found: ID does not exist" Jul 03 11:15:09.192007 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192001 2581 scope.go:117] "RemoveContainer" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" Jul 03 11:15:09.192224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192207 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c"} err="failed to get container status \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": rpc error: code = NotFound desc = could not find container \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": container with ID starting with a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c not found: ID does not exist" Jul 03 11:15:09.192270 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192226 2581 scope.go:117] "RemoveContainer" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" Jul 03 11:15:09.192470 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192454 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7"} err="failed to get container status \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": rpc error: code = NotFound desc = could not find container \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": container with ID starting with a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7 not found: ID does not exist" Jul 03 11:15:09.192517 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192470 2581 scope.go:117] "RemoveContainer" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" Jul 03 11:15:09.192685 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192669 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110"} err="failed to get container status \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": rpc error: code = NotFound desc = could not find container \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": container with ID starting with d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110 not found: ID does not exist" Jul 03 11:15:09.192732 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192686 2581 scope.go:117] "RemoveContainer" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" Jul 03 11:15:09.192900 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192870 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc"} err="failed to get container status \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": rpc error: code = NotFound desc = could not find container \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": container with ID starting with 038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc not found: ID does not exist" Jul 03 11:15:09.192900 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.192887 2581 scope.go:117] "RemoveContainer" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" Jul 03 11:15:09.193114 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.193093 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4"} err="failed to get container status \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": rpc error: code = NotFound desc = could not find container \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": container with ID starting with 30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4 not found: ID does not exist" Jul 03 11:15:09.193196 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.193115 2581 scope.go:117] "RemoveContainer" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" Jul 03 11:15:09.193399 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.193362 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d"} err="failed to get container status \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": rpc error: code = NotFound desc = could not find container \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": container with ID starting with ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d not found: ID does not exist" Jul 03 11:15:09.193399 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.193397 2581 scope.go:117] "RemoveContainer" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" Jul 03 11:15:09.193770 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.193745 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58"} err="failed to get container status \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": rpc error: code = NotFound desc = could not find container \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": container with ID starting with 3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58 not found: ID does not exist" Jul 03 11:15:09.193850 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.193773 2581 scope.go:117] "RemoveContainer" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" Jul 03 11:15:09.194028 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194009 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c"} err="failed to get container status \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": rpc error: code = NotFound desc = could not find container \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": container with ID starting with a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c not found: ID does not exist" Jul 03 11:15:09.194084 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194029 2581 scope.go:117] "RemoveContainer" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" Jul 03 11:15:09.194255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194236 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7"} err="failed to get container status \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": rpc error: code = NotFound desc = could not find container \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": container with ID starting with a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7 not found: ID does not exist" Jul 03 11:15:09.194299 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194257 2581 scope.go:117] "RemoveContainer" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" Jul 03 11:15:09.194577 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194547 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110"} err="failed to get container status \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": rpc error: code = NotFound desc = could not find container \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": container with ID starting with d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110 not found: ID does not exist" Jul 03 11:15:09.194577 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194566 2581 scope.go:117] "RemoveContainer" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" Jul 03 11:15:09.194773 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194758 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc"} err="failed to get container status \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": rpc error: code = NotFound desc = could not find container \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": container with ID starting with 038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc not found: ID does not exist" Jul 03 11:15:09.194813 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194775 2581 scope.go:117] "RemoveContainer" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" Jul 03 11:15:09.194993 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194974 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4"} err="failed to get container status \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": rpc error: code = NotFound desc = could not find container \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": container with ID starting with 30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4 not found: ID does not exist" Jul 03 11:15:09.194993 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.194994 2581 scope.go:117] "RemoveContainer" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" Jul 03 11:15:09.195180 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195164 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d"} err="failed to get container status \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": rpc error: code = NotFound desc = could not find container \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": container with ID starting with ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d not found: ID does not exist" Jul 03 11:15:09.195219 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195179 2581 scope.go:117] "RemoveContainer" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" Jul 03 11:15:09.195381 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195365 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58"} err="failed to get container status \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": rpc error: code = NotFound desc = could not find container \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": container with ID starting with 3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58 not found: ID does not exist" Jul 03 11:15:09.195419 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195381 2581 scope.go:117] "RemoveContainer" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" Jul 03 11:15:09.195611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195592 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c"} err="failed to get container status \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": rpc error: code = NotFound desc = could not find container \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": container with ID starting with a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c not found: ID does not exist" Jul 03 11:15:09.195660 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195611 2581 scope.go:117] "RemoveContainer" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" Jul 03 11:15:09.195816 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195799 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7"} err="failed to get container status \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": rpc error: code = NotFound desc = could not find container \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": container with ID starting with a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7 not found: ID does not exist" Jul 03 11:15:09.195854 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195817 2581 scope.go:117] "RemoveContainer" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" Jul 03 11:15:09.196001 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.195984 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110"} err="failed to get container status \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": rpc error: code = NotFound desc = could not find container \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": container with ID starting with d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110 not found: ID does not exist" Jul 03 11:15:09.196038 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196003 2581 scope.go:117] "RemoveContainer" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" Jul 03 11:15:09.196241 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196222 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc"} err="failed to get container status \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": rpc error: code = NotFound desc = could not find container \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": container with ID starting with 038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc not found: ID does not exist" Jul 03 11:15:09.196241 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196239 2581 scope.go:117] "RemoveContainer" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" Jul 03 11:15:09.196451 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196417 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4"} err="failed to get container status \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": rpc error: code = NotFound desc = could not find container \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": container with ID starting with 30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4 not found: ID does not exist" Jul 03 11:15:09.196451 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196443 2581 scope.go:117] "RemoveContainer" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" Jul 03 11:15:09.196662 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196641 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d"} err="failed to get container status \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": rpc error: code = NotFound desc = could not find container \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": container with ID starting with ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d not found: ID does not exist" Jul 03 11:15:09.196662 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196662 2581 scope.go:117] "RemoveContainer" containerID="3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58" Jul 03 11:15:09.196827 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196813 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58"} err="failed to get container status \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": rpc error: code = NotFound desc = could not find container \"3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58\": container with ID starting with 3f2207e0f2d46f620d4f8579bb3c8fddfbd5dabd5e22857233d179dac7f98e58 not found: ID does not exist" Jul 03 11:15:09.196882 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.196826 2581 scope.go:117] "RemoveContainer" containerID="a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c" Jul 03 11:15:09.197058 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197040 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c"} err="failed to get container status \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": rpc error: code = NotFound desc = could not find container \"a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c\": container with ID starting with a24b6fc5ed6bed5624fab1910c54e96040db254d93501860910902868bafe22c not found: ID does not exist" Jul 03 11:15:09.197100 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197059 2581 scope.go:117] "RemoveContainer" containerID="a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7" Jul 03 11:15:09.197258 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197243 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7"} err="failed to get container status \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": rpc error: code = NotFound desc = could not find container \"a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7\": container with ID starting with a6f2eaeccff60d7210ad97e99f31a7a8c8d2e7f4fdeebddaf7be65e8aa4708e7 not found: ID does not exist" Jul 03 11:15:09.197302 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197257 2581 scope.go:117] "RemoveContainer" containerID="d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110" Jul 03 11:15:09.197490 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197473 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110"} err="failed to get container status \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": rpc error: code = NotFound desc = could not find container \"d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110\": container with ID starting with d732ed3cb9dafac3c13b2b3152985bf1944df9d33cb245e598ce614141152110 not found: ID does not exist" Jul 03 11:15:09.197552 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197490 2581 scope.go:117] "RemoveContainer" containerID="038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc" Jul 03 11:15:09.197714 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197699 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc"} err="failed to get container status \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": rpc error: code = NotFound desc = could not find container \"038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc\": container with ID starting with 038ae918c9693595e733cd5f692774b4af33bfbe7d09338560f37ef8240f1afc not found: ID does not exist" Jul 03 11:15:09.197714 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197713 2581 scope.go:117] "RemoveContainer" containerID="30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4" Jul 03 11:15:09.197916 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197902 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4"} err="failed to get container status \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": rpc error: code = NotFound desc = could not find container \"30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4\": container with ID starting with 30504f74250bd4d893dd7ab05a21309678aa2930f34339c33a5cda597e889ff4 not found: ID does not exist" Jul 03 11:15:09.197954 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.197916 2581 scope.go:117] "RemoveContainer" containerID="ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d" Jul 03 11:15:09.198138 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.198120 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d"} err="failed to get container status \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": rpc error: code = NotFound desc = could not find container \"ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d\": container with ID starting with ae46fa8465e17e263a514a0a19f1eb115cc48ac482139ddafb1dd0d1f5b0a69d not found: ID does not exist" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.207138 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.207910 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="thanos-sidecar" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.207926 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="thanos-sidecar" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.207951 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy-web" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.207960 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy-web" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.207982 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.207991 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208028 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="config-reloader" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208036 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="config-reloader" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208054 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="init-config-reloader" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208063 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="init-config-reloader" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208073 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="prometheus" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208081 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="prometheus" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208103 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy-thanos" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208112 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy-thanos" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208255 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy-web" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208275 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="prometheus" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208286 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208302 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="thanos-sidecar" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208313 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="config-reloader" Jul 03 11:15:09.212139 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.208321 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" containerName="kube-rbac-proxy-thanos" Jul 03 11:15:09.215881 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.215841 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.218655 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.218632 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" Jul 03 11:15:09.218769 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.218633 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" Jul 03 11:15:09.218946 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.218930 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" Jul 03 11:15:09.219018 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.218941 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" Jul 03 11:15:09.219258 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.219240 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" Jul 03 11:15:09.219385 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.219265 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-bs1506rb0f1vo\"" Jul 03 11:15:09.219385 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.219277 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" Jul 03 11:15:09.219385 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.219305 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" Jul 03 11:15:09.219385 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.219247 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" Jul 03 11:15:09.219647 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.219630 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" Jul 03 11:15:09.219689 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.219663 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" Jul 03 11:15:09.219926 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.219892 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" Jul 03 11:15:09.220022 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.220007 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Jul 03 11:15:09.220504 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.220474 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" Jul 03 11:15:09.220504 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.220473 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-cfdtg\"" Jul 03 11:15:09.223106 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.223087 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" Jul 03 11:15:09.225931 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.225910 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" Jul 03 11:15:09.230470 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.230421 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:15:09.325938 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.325900 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326118 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.325956 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326118 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.325992 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326118 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326030 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326118 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326075 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326118 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326090 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326118 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326114 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f01678ac-2fa9-4e1c-9482-42ae450c6972-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326300 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326145 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326300 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326161 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326300 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326218 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326300 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326247 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-config\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326300 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326268 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326572 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326382 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326572 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326422 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326572 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326490 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326572 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326516 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-web-config\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326572 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326539 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-scr5d\" (UniqueName: \"kubernetes.io/projected/f01678ac-2fa9-4e1c-9482-42ae450c6972-kube-api-access-scr5d\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326572 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326567 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326775 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326618 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f01678ac-2fa9-4e1c-9482-42ae450c6972-config-out\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.326775 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.326659 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427347 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427360 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-config\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427497 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427567 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427594 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427623 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427645 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-web-config\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427666 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-scr5d\" (UniqueName: \"kubernetes.io/projected/f01678ac-2fa9-4e1c-9482-42ae450c6972-kube-api-access-scr5d\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427695 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427729 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f01678ac-2fa9-4e1c-9482-42ae450c6972-config-out\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427754 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427776 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.427834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427849 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427880 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427916 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427940 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.427972 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f01678ac-2fa9-4e1c-9482-42ae450c6972-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.428003 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.428026 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428565 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.428230 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.428971 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.428911 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.429150 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.429034 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.430229 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.429384 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.430643 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.430614 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.430875 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.430846 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-web-config\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.431125 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.431097 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.431216 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.431128 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.431562 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.431537 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.432051 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.432021 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.432137 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.432087 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.432137 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.432124 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-config\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.432332 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.432308 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.433096 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.433073 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.433417 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.433392 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.433513 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.433420 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f01678ac-2fa9-4e1c-9482-42ae450c6972-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.434001 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.433980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/f01678ac-2fa9-4e1c-9482-42ae450c6972-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.434470 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.434452 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/f01678ac-2fa9-4e1c-9482-42ae450c6972-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.435054 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.435030 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f01678ac-2fa9-4e1c-9482-42ae450c6972-config-out\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.438026 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.438001 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-scr5d\" (UniqueName: \"kubernetes.io/projected/f01678ac-2fa9-4e1c-9482-42ae450c6972-kube-api-access-scr5d\") pod \"prometheus-k8s-0\" (UID: \"f01678ac-2fa9-4e1c-9482-42ae450c6972\") " pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.527195 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.527158 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:09.671520 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.671489 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"] Jul 03 11:15:09.674634 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:15:09.674603 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf01678ac_2fa9_4e1c_9482_42ae450c6972.slice/crio-eea6ff0a50953e131e83a6bf2c457a768b068cabfec0cefa1cb5267c449364a9 WatchSource:0}: Error finding container eea6ff0a50953e131e83a6bf2c457a768b068cabfec0cefa1cb5267c449364a9: Status 404 returned error can't find the container with id eea6ff0a50953e131e83a6bf2c457a768b068cabfec0cefa1cb5267c449364a9 Jul 03 11:15:09.994867 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:09.994787 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60ba6df6-01ae-42d4-b45c-5b8b36d36f3c" path="/var/lib/kubelet/pods/60ba6df6-01ae-42d4-b45c-5b8b36d36f3c/volumes" Jul 03 11:15:10.141008 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:10.140977 2581 generic.go:356] "Generic (PLEG): container finished" podID="f01678ac-2fa9-4e1c-9482-42ae450c6972" containerID="f35139e95ed7a5233c0fc6a9ada92476f1cbf358618c250d362a44ec5237f641" exitCode=0 Jul 03 11:15:10.141154 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:10.141039 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f01678ac-2fa9-4e1c-9482-42ae450c6972","Type":"ContainerDied","Data":"f35139e95ed7a5233c0fc6a9ada92476f1cbf358618c250d362a44ec5237f641"} Jul 03 11:15:10.141154 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:10.141091 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f01678ac-2fa9-4e1c-9482-42ae450c6972","Type":"ContainerStarted","Data":"eea6ff0a50953e131e83a6bf2c457a768b068cabfec0cefa1cb5267c449364a9"} Jul 03 11:15:11.150652 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:11.150617 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f01678ac-2fa9-4e1c-9482-42ae450c6972","Type":"ContainerStarted","Data":"a6e4277a4b00c49cc4270cfdb2357a0aa1b7da94a1abcb4cfb216cc7108b9986"} Jul 03 11:15:11.151027 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:11.150679 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f01678ac-2fa9-4e1c-9482-42ae450c6972","Type":"ContainerStarted","Data":"11a8b786ecf560b0b78e23e4a003b94f5cd7c2237b863ff59f4ce14133c4c06d"} Jul 03 11:15:11.151027 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:11.150706 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f01678ac-2fa9-4e1c-9482-42ae450c6972","Type":"ContainerStarted","Data":"f922aa0d88a383227c66cfa494eec0db83c4b49f5ffb9e38e187d2e708caea82"} Jul 03 11:15:11.151027 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:11.150742 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f01678ac-2fa9-4e1c-9482-42ae450c6972","Type":"ContainerStarted","Data":"b5a8f96923ec701e3155ed802e9f2fac68561000c91feb73022a9e3960a37924"} Jul 03 11:15:11.151027 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:11.150771 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f01678ac-2fa9-4e1c-9482-42ae450c6972","Type":"ContainerStarted","Data":"fddeeac0a2151994c9d3483fae31e1d21b11987818fea78a197e5be8051d0e00"} Jul 03 11:15:11.151027 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:11.150796 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"f01678ac-2fa9-4e1c-9482-42ae450c6972","Type":"ContainerStarted","Data":"ace63301711bf9b87704c626a58ced253d75efbd679d796175b8921a4176caf2"} Jul 03 11:15:11.184668 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:11.184611 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.166442323 podStartE2EDuration="2.184590408s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.148085ms" isStatefulPod=false podCreationTimestamp="2026-07-03 11:15:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:15:11.182099168 +0000 UTC m=+173.744774680" watchObservedRunningTime="2026-07-03 11:15:11.184590408 +0000 UTC m=+173.747265983" Jul 03 11:15:14.527445 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:14.527395 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:15:24.042601 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:24.042561 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ddcc7f947-mqjf2"] Jul 03 11:15:49.072271 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.072195 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="openshift-console/console-ddcc7f947-mqjf2" podUID="f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" containerName="console" containerID="cri-o://2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa" gracePeriod=15 Jul 03 11:15:49.327359 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.327289 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:15:49.327763 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.327741 2581 generic.go:356] "Generic (PLEG): container finished" podID="f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" containerID="2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa" exitCode=0 Jul 03 11:15:49.327828 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.327812 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddcc7f947-mqjf2" event={"ID":"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c","Type":"ContainerDied","Data":"2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa"} Jul 03 11:15:49.327868 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.327860 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-ddcc7f947-mqjf2" event={"ID":"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c","Type":"ContainerDied","Data":"2a6a370d3255c2a27380414cd12e5e58fa742f0cf1b0c5e0ef62d99b8bbe7a96"} Jul 03 11:15:49.327900 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.327880 2581 scope.go:117] "RemoveContainer" containerID="2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa" Jul 03 11:15:49.336236 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.336219 2581 scope.go:117] "RemoveContainer" containerID="2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa" Jul 03 11:15:49.336513 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:15:49.336495 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa\": container with ID starting with 2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa not found: ID does not exist" containerID="2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa" Jul 03 11:15:49.336582 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.336522 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa"} err="failed to get container status \"2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa\": rpc error: code = NotFound desc = could not find container \"2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa\": container with ID starting with 2d13fe4cde43da2bb9b5b0043d6a134da206498105c589bc1805f1d2bbd61ffa not found: ID does not exist" Jul 03 11:15:49.475404 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475346 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-serving-cert\") pod \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " Jul 03 11:15:49.475404 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475398 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-config\") pod \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " Jul 03 11:15:49.475658 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475455 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cn2kn\" (UniqueName: \"kubernetes.io/projected/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-kube-api-access-cn2kn\") pod \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " Jul 03 11:15:49.475658 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475519 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-oauth-config\") pod \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " Jul 03 11:15:49.475658 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475548 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-service-ca\") pod \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " Jul 03 11:15:49.475658 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475599 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-trusted-ca-bundle\") pod \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " Jul 03 11:15:49.475658 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475623 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-oauth-serving-cert\") pod \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\" (UID: \"f3046ab3-ff2a-4a03-8e94-1ee948b4a76c\") " Jul 03 11:15:49.475942 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475913 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-config" (OuterVolumeSpecName: "console-config") pod "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" (UID: "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:49.476024 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.475988 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-service-ca" (OuterVolumeSpecName: "service-ca") pod "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" (UID: "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:49.476076 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.476021 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" (UID: "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:49.476076 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.476034 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" (UID: "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:15:49.478046 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.478008 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-kube-api-access-cn2kn" (OuterVolumeSpecName: "kube-api-access-cn2kn") pod "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" (UID: "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c"). InnerVolumeSpecName "kube-api-access-cn2kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:15:49.478046 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.478007 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" (UID: "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:49.478188 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.478080 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" (UID: "f3046ab3-ff2a-4a03-8e94-1ee948b4a76c"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:15:49.577087 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.577056 2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-serving-cert\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:49.577087 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.577086 2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-config\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:49.577281 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.577101 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cn2kn\" (UniqueName: \"kubernetes.io/projected/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-kube-api-access-cn2kn\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:49.577281 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.577117 2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-console-oauth-config\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:49.577281 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.577132 2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-service-ca\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:49.577281 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.577143 2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-trusted-ca-bundle\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:49.577281 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:49.577152 2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c-oauth-serving-cert\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:15:50.332479 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:50.332376 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-ddcc7f947-mqjf2" Jul 03 11:15:50.350776 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:50.350744 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-ddcc7f947-mqjf2"] Jul 03 11:15:50.354877 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:50.354850 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-ddcc7f947-mqjf2"] Jul 03 11:15:51.993658 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:15:51.993624 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" path="/var/lib/kubelet/pods/f3046ab3-ff2a-4a03-8e94-1ee948b4a76c/volumes" Jul 03 11:16:09.527632 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:16:09.527588 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:16:09.544657 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:16:09.544628 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:16:10.432856 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:16:10.432825 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0" Jul 03 11:17:17.864361 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:17:17.864324 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:17:17.870907 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:17:17.870881 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:17:17.872616 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:17:17.872591 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:17:17.876735 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:17:17.876713 2581 kubelet.go:1648] "Image garbage collection succeeded" Jul 03 11:17:17.879088 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:17:17.879069 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:18:43.089508 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.089471 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d"] Jul 03 11:18:43.090069 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.089894 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" containerName="console" Jul 03 11:18:43.090069 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.089907 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" containerName="console" Jul 03 11:18:43.090069 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.089970 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3046ab3-ff2a-4a03-8e94-1ee948b4a76c" containerName="console" Jul 03 11:18:43.093172 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.093148 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.096924 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.096900 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Jul 03 11:18:43.097051 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.096908 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Jul 03 11:18:43.097349 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.097333 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-nbwwk\"" Jul 03 11:18:43.097525 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.097511 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Jul 03 11:18:43.102688 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.102671 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Jul 03 11:18:43.125509 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.125475 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d"] Jul 03 11:18:43.170408 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.170364 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a523fa-70f8-48bf-8ad3-5ca0575733e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.170408 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.170410 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9lnh\" (UniqueName: \"kubernetes.io/projected/65a523fa-70f8-48bf-8ad3-5ca0575733e8-kube-api-access-d9lnh\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.170679 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.170504 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65a523fa-70f8-48bf-8ad3-5ca0575733e8-tmp\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.170679 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.170528 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65a523fa-70f8-48bf-8ad3-5ca0575733e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.170679 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.170596 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65a523fa-70f8-48bf-8ad3-5ca0575733e8-manifests\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.271617 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.271571 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a523fa-70f8-48bf-8ad3-5ca0575733e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.271617 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.271613 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9lnh\" (UniqueName: \"kubernetes.io/projected/65a523fa-70f8-48bf-8ad3-5ca0575733e8-kube-api-access-d9lnh\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.271871 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.271644 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65a523fa-70f8-48bf-8ad3-5ca0575733e8-tmp\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.271871 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.271660 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65a523fa-70f8-48bf-8ad3-5ca0575733e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.271871 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.271848 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65a523fa-70f8-48bf-8ad3-5ca0575733e8-manifests\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.272184 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.272154 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/65a523fa-70f8-48bf-8ad3-5ca0575733e8-manifests\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.274213 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.274187 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/65a523fa-70f8-48bf-8ad3-5ca0575733e8-tmp\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.274351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.274253 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/65a523fa-70f8-48bf-8ad3-5ca0575733e8-apiservice-cert\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.274407 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.274354 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/65a523fa-70f8-48bf-8ad3-5ca0575733e8-webhook-cert\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.280864 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.280842 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9lnh\" (UniqueName: \"kubernetes.io/projected/65a523fa-70f8-48bf-8ad3-5ca0575733e8-kube-api-access-d9lnh\") pod \"opendatahub-operator-controller-manager-b8c56cd69-ftb5d\" (UID: \"65a523fa-70f8-48bf-8ad3-5ca0575733e8\") " pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.404547 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.404517 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:43.558010 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.557985 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d"] Jul 03 11:18:43.561068 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:18:43.561040 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65a523fa_70f8_48bf_8ad3_5ca0575733e8.slice/crio-90dd857e737da6bdb488244811a3b29f9809e739fc661d8c0875ca85aad47352 WatchSource:0}: Error finding container 90dd857e737da6bdb488244811a3b29f9809e739fc661d8c0875ca85aad47352: Status 404 returned error can't find the container with id 90dd857e737da6bdb488244811a3b29f9809e739fc661d8c0875ca85aad47352 Jul 03 11:18:43.562938 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:43.562919 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 11:18:44.063581 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:44.063545 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" event={"ID":"65a523fa-70f8-48bf-8ad3-5ca0575733e8","Type":"ContainerStarted","Data":"90dd857e737da6bdb488244811a3b29f9809e739fc661d8c0875ca85aad47352"} Jul 03 11:18:47.080580 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:47.080540 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" event={"ID":"65a523fa-70f8-48bf-8ad3-5ca0575733e8","Type":"ContainerStarted","Data":"0da715b45045b2ab4cfdc5461d96284bf8eb5b32cca8fb766784421c5f6f4da7"} Jul 03 11:18:48.086789 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:48.086689 2581 generic.go:356] "Generic (PLEG): container finished" podID="65a523fa-70f8-48bf-8ad3-5ca0575733e8" containerID="0da715b45045b2ab4cfdc5461d96284bf8eb5b32cca8fb766784421c5f6f4da7" exitCode=0 Jul 03 11:18:48.087167 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:48.086787 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" event={"ID":"65a523fa-70f8-48bf-8ad3-5ca0575733e8","Type":"ContainerDied","Data":"0da715b45045b2ab4cfdc5461d96284bf8eb5b32cca8fb766784421c5f6f4da7"} Jul 03 11:18:49.092784 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:49.092753 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" event={"ID":"65a523fa-70f8-48bf-8ad3-5ca0575733e8","Type":"ContainerStarted","Data":"9f74cd74a596933b71b945018b1d136348a1d704d858b48f88662a81c60f18a3"} Jul 03 11:18:49.093174 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:49.092854 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:18:49.116488 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:18:49.116414 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" podStartSLOduration=1.562253699 podStartE2EDuration="6.116397065s" totalImagesPullingTime="3.392931931s" totalInitContainerRuntime="1.161211435s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:18:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:18:49.114264459 +0000 UTC m=+391.676939970" watchObservedRunningTime="2026-07-03 11:18:49.116397065 +0000 UTC m=+391.679072577" Jul 03 11:19:00.100564 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:00.100534 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-b8c56cd69-ftb5d" Jul 03 11:19:02.550686 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.550649 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2"] Jul 03 11:19:02.561334 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.561305 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.566351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.566327 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Jul 03 11:19:02.566524 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.566478 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Jul 03 11:19:02.567503 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.567483 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Jul 03 11:19:02.567503 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.567494 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fnjxq\"" Jul 03 11:19:02.567698 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.567679 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Jul 03 11:19:02.569671 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.569653 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Jul 03 11:19:02.583963 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.583935 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2"] Jul 03 11:19:02.744055 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.744018 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-cert\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.744233 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.744069 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvbt6\" (UniqueName: \"kubernetes.io/projected/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-kube-api-access-vvbt6\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.744233 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.744128 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-manager-config\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.744233 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.744191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-metrics-cert\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.845270 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.845178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-metrics-cert\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.845270 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.845247 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-cert\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.845546 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.845279 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvbt6\" (UniqueName: \"kubernetes.io/projected/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-kube-api-access-vvbt6\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.845546 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.845299 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-manager-config\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.846107 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.846070 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-manager-config\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.848006 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.847984 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-cert\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.848114 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.848018 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-metrics-cert\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.864193 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.864166 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvbt6\" (UniqueName: \"kubernetes.io/projected/d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f-kube-api-access-vvbt6\") pod \"lws-controller-manager-6f9f46b759-sblp2\" (UID: \"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f\") " pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:02.871023 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:02.871005 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:03.025561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.025532 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2"] Jul 03 11:19:03.027725 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:19:03.027698 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2d9a0d9_f891_46e4_9578_b4a2e1e4a85f.slice/crio-65521df58a401379c366047642e26507a2bfb16ce9b941f80bf4567fcb810c27 WatchSource:0}: Error finding container 65521df58a401379c366047642e26507a2bfb16ce9b941f80bf4567fcb810c27: Status 404 returned error can't find the container with id 65521df58a401379c366047642e26507a2bfb16ce9b941f80bf4567fcb810c27 Jul 03 11:19:03.161052 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.161015 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" event={"ID":"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f","Type":"ContainerStarted","Data":"65521df58a401379c366047642e26507a2bfb16ce9b941f80bf4567fcb810c27"} Jul 03 11:19:03.197257 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.197224 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d"] Jul 03 11:19:03.200761 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.200740 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.204804 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.204777 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Jul 03 11:19:03.205788 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.205769 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wtv2j\"" Jul 03 11:19:03.205887 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.205822 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Jul 03 11:19:03.223633 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.223595 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d"] Jul 03 11:19:03.349837 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.349798 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-tls-certs\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.350029 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.349847 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-tmp\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.350029 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.349948 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9h7nd\" (UniqueName: \"kubernetes.io/projected/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-kube-api-access-9h7nd\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.450672 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.450575 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-tls-certs\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.450672 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.450622 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-tmp\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.450672 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.450665 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9h7nd\" (UniqueName: \"kubernetes.io/projected/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-kube-api-access-9h7nd\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.453133 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.453107 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-tmp\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.453328 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.453310 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-tls-certs\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.459476 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.459452 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9h7nd\" (UniqueName: \"kubernetes.io/projected/04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5-kube-api-access-9h7nd\") pod \"kube-auth-proxy-5cb49b9d79-vhj2d\" (UID: \"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5\") " pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.512338 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.512305 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" Jul 03 11:19:03.665496 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:03.665319 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d"] Jul 03 11:19:03.668556 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:19:03.668476 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04c0c892_9e63_4a3d_8c9a_fc8e87c5f8f5.slice/crio-ea980542dacbcb5bd4479d8a911fd590c7844575b7c64a791aa4170ecef02146 WatchSource:0}: Error finding container ea980542dacbcb5bd4479d8a911fd590c7844575b7c64a791aa4170ecef02146: Status 404 returned error can't find the container with id ea980542dacbcb5bd4479d8a911fd590c7844575b7c64a791aa4170ecef02146 Jul 03 11:19:04.168691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:04.168652 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" event={"ID":"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5","Type":"ContainerStarted","Data":"ea980542dacbcb5bd4479d8a911fd590c7844575b7c64a791aa4170ecef02146"} Jul 03 11:19:06.183692 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:06.183646 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" event={"ID":"d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f","Type":"ContainerStarted","Data":"b3ab46273e9f4a109cd6c00b8f99882a2b48aeac5e220d25ba926a0ed54326c9"} Jul 03 11:19:06.184163 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:06.183749 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:19:06.203632 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:06.203581 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" podStartSLOduration=1.848387937 podStartE2EDuration="4.203561415s" totalImagesPullingTime="2.355173478s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:19:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:19:06.202481817 +0000 UTC m=+408.765157335" watchObservedRunningTime="2026-07-03 11:19:06.203561415 +0000 UTC m=+408.766236927" Jul 03 11:19:07.189591 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:07.189549 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" event={"ID":"04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5","Type":"ContainerStarted","Data":"6f23a027addb967d86fa1c013863962880180e3eacf05a59201b24f6e8674b2e"} Jul 03 11:19:07.208948 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:07.208884 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5cb49b9d79-vhj2d" podStartSLOduration=0.868357552 podStartE2EDuration="4.208866725s" totalImagesPullingTime="3.340509173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:19:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:19:07.206855101 +0000 UTC m=+409.769530614" watchObservedRunningTime="2026-07-03 11:19:07.208866725 +0000 UTC m=+409.771542237" Jul 03 11:19:17.191805 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:19:17.191769 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6f9f46b759-sblp2" Jul 03 11:20:57.456446 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.456396 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk"] Jul 03 11:20:57.459890 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.459865 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:57.462569 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.462535 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Jul 03 11:20:57.463146 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.463108 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Jul 03 11:20:57.471067 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.471025 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Jul 03 11:20:57.471587 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.471048 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Jul 03 11:20:57.471736 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.471544 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wfhwf\"" Jul 03 11:20:57.473905 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.473877 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk"] Jul 03 11:20:57.574858 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.574823 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3b5b04-b58e-480f-9ed1-b976724346b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:57.575041 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.574886 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq96r\" (UniqueName: \"kubernetes.io/projected/0b3b5b04-b58e-480f-9ed1-b976724346b2-kube-api-access-fq96r\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:57.575041 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.574937 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b3b5b04-b58e-480f-9ed1-b976724346b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:57.675747 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.675701 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq96r\" (UniqueName: \"kubernetes.io/projected/0b3b5b04-b58e-480f-9ed1-b976724346b2-kube-api-access-fq96r\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:57.675950 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.675762 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b3b5b04-b58e-480f-9ed1-b976724346b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:57.675950 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.675838 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3b5b04-b58e-480f-9ed1-b976724346b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:57.676078 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:20:57.675950 2581 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found Jul 03 11:20:57.676078 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:20:57.676054 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0b3b5b04-b58e-480f-9ed1-b976724346b2-plugin-serving-cert podName:0b3b5b04-b58e-480f-9ed1-b976724346b2 nodeName:}" failed. No retries permitted until 2026-07-03 11:20:58.176013704 +0000 UTC m=+520.738689198 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/0b3b5b04-b58e-480f-9ed1-b976724346b2-plugin-serving-cert") pod "kuadrant-console-plugin-66b5774985-lfsdk" (UID: "0b3b5b04-b58e-480f-9ed1-b976724346b2") : secret "plugin-serving-cert" not found Jul 03 11:20:57.676471 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.676418 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0b3b5b04-b58e-480f-9ed1-b976724346b2-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:57.686026 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:57.685997 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq96r\" (UniqueName: \"kubernetes.io/projected/0b3b5b04-b58e-480f-9ed1-b976724346b2-kube-api-access-fq96r\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:58.178813 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:58.178776 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3b5b04-b58e-480f-9ed1-b976724346b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:58.181475 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:58.181421 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/0b3b5b04-b58e-480f-9ed1-b976724346b2-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-lfsdk\" (UID: \"0b3b5b04-b58e-480f-9ed1-b976724346b2\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:58.390504 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:58.390462 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" Jul 03 11:20:58.540621 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:58.540590 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk"] Jul 03 11:20:58.541772 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:20:58.541746 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0b3b5b04_b58e_480f_9ed1_b976724346b2.slice/crio-d638d1a3e6d76b2e558fe7ac4f0b3bee4f31bee255e7cd698c8ee5a8edea8abf WatchSource:0}: Error finding container d638d1a3e6d76b2e558fe7ac4f0b3bee4f31bee255e7cd698c8ee5a8edea8abf: Status 404 returned error can't find the container with id d638d1a3e6d76b2e558fe7ac4f0b3bee4f31bee255e7cd698c8ee5a8edea8abf Jul 03 11:20:58.698558 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:20:58.698471 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" event={"ID":"0b3b5b04-b58e-480f-9ed1-b976724346b2","Type":"ContainerStarted","Data":"d638d1a3e6d76b2e558fe7ac4f0b3bee4f31bee255e7cd698c8ee5a8edea8abf"} Jul 03 11:21:20.829415 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:20.829327 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" event={"ID":"0b3b5b04-b58e-480f-9ed1-b976724346b2","Type":"ContainerStarted","Data":"417bf650c4ca36af36a44b7fa70d4bf5cd3290de0abd889e8618b52a28a3fba5"} Jul 03 11:21:20.852146 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:20.852091 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-lfsdk" podStartSLOduration=1.9214825260000001 podStartE2EDuration="23.852074063s" totalImagesPullingTime="21.930591537s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:20:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:21:20.849814024 +0000 UTC m=+543.412489541" watchObservedRunningTime="2026-07-03 11:21:20.852074063 +0000 UTC m=+543.414749575" Jul 03 11:21:38.048297 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.048263 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7pjsz"] Jul 03 11:21:38.068637 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.068600 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7pjsz"] Jul 03 11:21:38.068825 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.068725 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:38.075583 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.075545 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Jul 03 11:21:38.138906 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.138869 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7pjsz"] Jul 03 11:21:38.165421 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.165388 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgf5k\" (UniqueName: \"kubernetes.io/projected/114fb8db-5e29-426a-83ff-803794122013-kube-api-access-pgf5k\") pod \"limitador-limitador-64c94fd64f-7pjsz\" (UID: \"114fb8db-5e29-426a-83ff-803794122013\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:38.165615 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.165460 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/114fb8db-5e29-426a-83ff-803794122013-config-file\") pod \"limitador-limitador-64c94fd64f-7pjsz\" (UID: \"114fb8db-5e29-426a-83ff-803794122013\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:38.266855 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.266811 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgf5k\" (UniqueName: \"kubernetes.io/projected/114fb8db-5e29-426a-83ff-803794122013-kube-api-access-pgf5k\") pod \"limitador-limitador-64c94fd64f-7pjsz\" (UID: \"114fb8db-5e29-426a-83ff-803794122013\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:38.266855 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.266856 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/114fb8db-5e29-426a-83ff-803794122013-config-file\") pod \"limitador-limitador-64c94fd64f-7pjsz\" (UID: \"114fb8db-5e29-426a-83ff-803794122013\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:38.267526 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.267509 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/114fb8db-5e29-426a-83ff-803794122013-config-file\") pod \"limitador-limitador-64c94fd64f-7pjsz\" (UID: \"114fb8db-5e29-426a-83ff-803794122013\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:38.275687 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.275652 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgf5k\" (UniqueName: \"kubernetes.io/projected/114fb8db-5e29-426a-83ff-803794122013-kube-api-access-pgf5k\") pod \"limitador-limitador-64c94fd64f-7pjsz\" (UID: \"114fb8db-5e29-426a-83ff-803794122013\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:38.381165 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.381073 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:38.548781 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:21:38.548746 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod114fb8db_5e29_426a_83ff_803794122013.slice/crio-f07fbdb7462155922c799a76661d64abb19557b146c0979c7ce7a66068b19913 WatchSource:0}: Error finding container f07fbdb7462155922c799a76661d64abb19557b146c0979c7ce7a66068b19913: Status 404 returned error can't find the container with id f07fbdb7462155922c799a76661d64abb19557b146c0979c7ce7a66068b19913 Jul 03 11:21:38.549146 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.549122 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7pjsz"] Jul 03 11:21:38.920380 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:38.920343 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" event={"ID":"114fb8db-5e29-426a-83ff-803794122013","Type":"ContainerStarted","Data":"f07fbdb7462155922c799a76661d64abb19557b146c0979c7ce7a66068b19913"} Jul 03 11:21:41.938448 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:41.938398 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" event={"ID":"114fb8db-5e29-426a-83ff-803794122013","Type":"ContainerStarted","Data":"9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da"} Jul 03 11:21:41.938864 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:41.938478 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:41.956470 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:41.956395 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" podStartSLOduration=0.687035144 podStartE2EDuration="3.9563764s" totalImagesPullingTime="3.269341256s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:21:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:21:41.955140577 +0000 UTC m=+564.517816090" watchObservedRunningTime="2026-07-03 11:21:41.9563764 +0000 UTC m=+564.519051913" Jul 03 11:21:52.944128 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:52.944096 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:53.204758 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.204674 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7pjsz"] Jul 03 11:21:53.204910 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.204889 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" podUID="114fb8db-5e29-426a-83ff-803794122013" containerName="limitador" containerID="cri-o://9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da" gracePeriod=30 Jul 03 11:21:53.790057 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.790025 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:53.911237 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.911201 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/114fb8db-5e29-426a-83ff-803794122013-config-file\") pod \"114fb8db-5e29-426a-83ff-803794122013\" (UID: \"114fb8db-5e29-426a-83ff-803794122013\") " Jul 03 11:21:53.911426 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.911291 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pgf5k\" (UniqueName: \"kubernetes.io/projected/114fb8db-5e29-426a-83ff-803794122013-kube-api-access-pgf5k\") pod \"114fb8db-5e29-426a-83ff-803794122013\" (UID: \"114fb8db-5e29-426a-83ff-803794122013\") " Jul 03 11:21:53.911611 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.911587 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/114fb8db-5e29-426a-83ff-803794122013-config-file" (OuterVolumeSpecName: "config-file") pod "114fb8db-5e29-426a-83ff-803794122013" (UID: "114fb8db-5e29-426a-83ff-803794122013"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Jul 03 11:21:53.913630 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.913602 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/114fb8db-5e29-426a-83ff-803794122013-kube-api-access-pgf5k" (OuterVolumeSpecName: "kube-api-access-pgf5k") pod "114fb8db-5e29-426a-83ff-803794122013" (UID: "114fb8db-5e29-426a-83ff-803794122013"). InnerVolumeSpecName "kube-api-access-pgf5k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:21:53.997455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.997391 2581 generic.go:356] "Generic (PLEG): container finished" podID="114fb8db-5e29-426a-83ff-803794122013" containerID="9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da" exitCode=0 Jul 03 11:21:53.997455 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.997454 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" Jul 03 11:21:53.997982 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.997499 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" event={"ID":"114fb8db-5e29-426a-83ff-803794122013","Type":"ContainerDied","Data":"9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da"} Jul 03 11:21:53.997982 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.997560 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-7pjsz" event={"ID":"114fb8db-5e29-426a-83ff-803794122013","Type":"ContainerDied","Data":"f07fbdb7462155922c799a76661d64abb19557b146c0979c7ce7a66068b19913"} Jul 03 11:21:53.997982 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:53.997594 2581 scope.go:117] "RemoveContainer" containerID="9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da" Jul 03 11:21:54.006974 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:54.006955 2581 scope.go:117] "RemoveContainer" containerID="9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da" Jul 03 11:21:54.007278 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:21:54.007261 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da\": container with ID starting with 9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da not found: ID does not exist" containerID="9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da" Jul 03 11:21:54.007350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:54.007286 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da"} err="failed to get container status \"9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da\": rpc error: code = NotFound desc = could not find container \"9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da\": container with ID starting with 9fc1e8fb88d324dd6e224ed4c8cfddd1c4b5281260f5934c4ee0e3c62370e0da not found: ID does not exist" Jul 03 11:21:54.012768 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:54.012746 2581 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/114fb8db-5e29-426a-83ff-803794122013-config-file\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:21:54.012768 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:54.012771 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgf5k\" (UniqueName: \"kubernetes.io/projected/114fb8db-5e29-426a-83ff-803794122013-kube-api-access-pgf5k\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:21:54.021962 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:54.021885 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7pjsz"] Jul 03 11:21:54.026409 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:54.026380 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-7pjsz"] Jul 03 11:21:55.994812 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:55.994776 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="114fb8db-5e29-426a-83ff-803794122013" path="/var/lib/kubelet/pods/114fb8db-5e29-426a-83ff-803794122013/volumes" Jul 03 11:21:59.590843 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.590809 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/postgres-69fb4594cf-9hpmt"] Jul 03 11:21:59.591255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.591184 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="114fb8db-5e29-426a-83ff-803794122013" containerName="limitador" Jul 03 11:21:59.591255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.591196 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="114fb8db-5e29-426a-83ff-803794122013" containerName="limitador" Jul 03 11:21:59.591339 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.591280 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="114fb8db-5e29-426a-83ff-803794122013" containerName="limitador" Jul 03 11:21:59.595857 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.595834 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:21:59.598691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.598667 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"postgres-creds\"" Jul 03 11:21:59.598847 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.598730 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-bbklm\"" Jul 03 11:21:59.607026 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.606997 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-9hpmt"] Jul 03 11:21:59.765267 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.765219 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/873236fb-b94a-4729-acf6-ada0a1fa5133-data\") pod \"postgres-69fb4594cf-9hpmt\" (UID: \"873236fb-b94a-4729-acf6-ada0a1fa5133\") " pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:21:59.765267 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.765264 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khgh6\" (UniqueName: \"kubernetes.io/projected/873236fb-b94a-4729-acf6-ada0a1fa5133-kube-api-access-khgh6\") pod \"postgres-69fb4594cf-9hpmt\" (UID: \"873236fb-b94a-4729-acf6-ada0a1fa5133\") " pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:21:59.866650 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.866558 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/873236fb-b94a-4729-acf6-ada0a1fa5133-data\") pod \"postgres-69fb4594cf-9hpmt\" (UID: \"873236fb-b94a-4729-acf6-ada0a1fa5133\") " pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:21:59.866650 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.866608 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khgh6\" (UniqueName: \"kubernetes.io/projected/873236fb-b94a-4729-acf6-ada0a1fa5133-kube-api-access-khgh6\") pod \"postgres-69fb4594cf-9hpmt\" (UID: \"873236fb-b94a-4729-acf6-ada0a1fa5133\") " pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:21:59.867014 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.866996 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/873236fb-b94a-4729-acf6-ada0a1fa5133-data\") pod \"postgres-69fb4594cf-9hpmt\" (UID: \"873236fb-b94a-4729-acf6-ada0a1fa5133\") " pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:21:59.876776 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.876749 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khgh6\" (UniqueName: \"kubernetes.io/projected/873236fb-b94a-4729-acf6-ada0a1fa5133-kube-api-access-khgh6\") pod \"postgres-69fb4594cf-9hpmt\" (UID: \"873236fb-b94a-4729-acf6-ada0a1fa5133\") " pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:21:59.907758 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:21:59.907722 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:22:00.049474 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:00.049405 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/postgres-69fb4594cf-9hpmt"] Jul 03 11:22:00.053496 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:22:00.053470 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod873236fb_b94a_4729_acf6_ada0a1fa5133.slice/crio-5f8c2b83339f815d4d55d8b1bba86584740ca78e51327948b81deb2cc2533231 WatchSource:0}: Error finding container 5f8c2b83339f815d4d55d8b1bba86584740ca78e51327948b81deb2cc2533231: Status 404 returned error can't find the container with id 5f8c2b83339f815d4d55d8b1bba86584740ca78e51327948b81deb2cc2533231 Jul 03 11:22:01.037416 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:01.037288 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-9hpmt" event={"ID":"873236fb-b94a-4729-acf6-ada0a1fa5133","Type":"ContainerStarted","Data":"5f8c2b83339f815d4d55d8b1bba86584740ca78e51327948b81deb2cc2533231"} Jul 03 11:22:05.058881 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:05.058844 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/postgres-69fb4594cf-9hpmt" event={"ID":"873236fb-b94a-4729-acf6-ada0a1fa5133","Type":"ContainerStarted","Data":"848c5313749ad409f5ea446ca307f00700d679c65d1bbcc7b3db9a5abf57db48"} Jul 03 11:22:05.059401 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:05.058968 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:22:05.079476 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:05.079401 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/postgres-69fb4594cf-9hpmt" podStartSLOduration=1.279352364 podStartE2EDuration="6.079384379s" totalImagesPullingTime="4.800032015s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:21:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:22:05.077736415 +0000 UTC m=+587.640411928" watchObservedRunningTime="2026-07-03 11:22:05.079384379 +0000 UTC m=+587.642059890" Jul 03 11:22:11.095444 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:11.095400 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/postgres-69fb4594cf-9hpmt" Jul 03 11:22:17.570751 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.570710 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7588b74558-xf6hd"] Jul 03 11:22:17.579494 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.579467 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:17.582975 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.582953 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 03 11:22:17.583172 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.582952 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mvflh\"" Jul 03 11:22:17.583945 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.583912 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 03 11:22:17.586976 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.586951 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7588b74558-xf6hd"] Jul 03 11:22:17.623340 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.623300 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/b52e3d1c-2f76-459e-af36-526b41639c9c-kube-api-access-flzcg\") pod \"maas-controller-7588b74558-xf6hd\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:17.623531 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.623369 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert\") pod \"maas-controller-7588b74558-xf6hd\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:17.724225 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.724178 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/b52e3d1c-2f76-459e-af36-526b41639c9c-kube-api-access-flzcg\") pod \"maas-controller-7588b74558-xf6hd\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:17.724415 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.724265 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert\") pod \"maas-controller-7588b74558-xf6hd\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:17.724415 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:22:17.724395 2581 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found Jul 03 11:22:17.724515 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:22:17.724482 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert podName:b52e3d1c-2f76-459e-af36-526b41639c9c nodeName:}" failed. No retries permitted until 2026-07-03 11:22:18.224463398 +0000 UTC m=+600.787138888 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert") pod "maas-controller-7588b74558-xf6hd" (UID: "b52e3d1c-2f76-459e-af36-526b41639c9c") : secret "maas-controller-webhook-cert" not found Jul 03 11:22:17.742795 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.742757 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/b52e3d1c-2f76-459e-af36-526b41639c9c-kube-api-access-flzcg\") pod \"maas-controller-7588b74558-xf6hd\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:17.898508 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.898483 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:22:17.900560 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.900534 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:22:17.905776 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.905752 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:22:17.908009 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:17.907991 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:22:18.228319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:18.228215 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert\") pod \"maas-controller-7588b74558-xf6hd\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:18.230806 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:18.230776 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert\") pod \"maas-controller-7588b74558-xf6hd\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:18.496416 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:18.496326 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mvflh\"" Jul 03 11:22:18.503731 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:18.503704 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:18.649171 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:18.649143 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7588b74558-xf6hd"] Jul 03 11:22:18.651582 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:22:18.651556 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52e3d1c_2f76_459e_af36_526b41639c9c.slice/crio-55cbcddcaf71a8a562142db6823a5025be84f22a4c0dcad6c2abb9b13178f9aa WatchSource:0}: Error finding container 55cbcddcaf71a8a562142db6823a5025be84f22a4c0dcad6c2abb9b13178f9aa: Status 404 returned error can't find the container with id 55cbcddcaf71a8a562142db6823a5025be84f22a4c0dcad6c2abb9b13178f9aa Jul 03 11:22:19.128315 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:19.128274 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7588b74558-xf6hd" event={"ID":"b52e3d1c-2f76-459e-af36-526b41639c9c","Type":"ContainerStarted","Data":"55cbcddcaf71a8a562142db6823a5025be84f22a4c0dcad6c2abb9b13178f9aa"} Jul 03 11:22:21.135420 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:21.135390 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 03 11:22:22.144604 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:22.144568 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7588b74558-xf6hd" event={"ID":"b52e3d1c-2f76-459e-af36-526b41639c9c","Type":"ContainerStarted","Data":"765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9"} Jul 03 11:22:22.145005 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:22.144665 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:22.163170 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:22.163115 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7588b74558-xf6hd" podStartSLOduration=2.684263752 podStartE2EDuration="5.163097363s" totalImagesPullingTime="2.478833611s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:22:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:22:22.16270637 +0000 UTC m=+604.725381882" watchObservedRunningTime="2026-07-03 11:22:22.163097363 +0000 UTC m=+604.725772875" Jul 03 11:22:32.315154 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.315119 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7588b74558-xf6hd"] Jul 03 11:22:32.315686 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.315554 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="opendatahub/maas-controller-7588b74558-xf6hd" podUID="b52e3d1c-2f76-459e-af36-526b41639c9c" containerName="manager" containerID="cri-o://765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9" gracePeriod=30 Jul 03 11:22:32.323871 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.323827 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:32.560957 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.560931 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:32.667669 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.667629 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/b52e3d1c-2f76-459e-af36-526b41639c9c-kube-api-access-flzcg\") pod \"b52e3d1c-2f76-459e-af36-526b41639c9c\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " Jul 03 11:22:32.667853 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.667740 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert\") pod \"b52e3d1c-2f76-459e-af36-526b41639c9c\" (UID: \"b52e3d1c-2f76-459e-af36-526b41639c9c\") " Jul 03 11:22:32.670045 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.670014 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "b52e3d1c-2f76-459e-af36-526b41639c9c" (UID: "b52e3d1c-2f76-459e-af36-526b41639c9c"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:22:32.670172 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.670045 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52e3d1c-2f76-459e-af36-526b41639c9c-kube-api-access-flzcg" (OuterVolumeSpecName: "kube-api-access-flzcg") pod "b52e3d1c-2f76-459e-af36-526b41639c9c" (UID: "b52e3d1c-2f76-459e-af36-526b41639c9c"). InnerVolumeSpecName "kube-api-access-flzcg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:22:32.768984 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.768938 2581 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b52e3d1c-2f76-459e-af36-526b41639c9c-webhook-cert\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:22:32.768984 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:32.768975 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flzcg\" (UniqueName: \"kubernetes.io/projected/b52e3d1c-2f76-459e-af36-526b41639c9c-kube-api-access-flzcg\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:22:33.198180 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.198143 2581 generic.go:356] "Generic (PLEG): container finished" podID="b52e3d1c-2f76-459e-af36-526b41639c9c" containerID="765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9" exitCode=0 Jul 03 11:22:33.198363 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.198192 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7588b74558-xf6hd" Jul 03 11:22:33.198363 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.198225 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7588b74558-xf6hd" event={"ID":"b52e3d1c-2f76-459e-af36-526b41639c9c","Type":"ContainerDied","Data":"765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9"} Jul 03 11:22:33.198363 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.198280 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7588b74558-xf6hd" event={"ID":"b52e3d1c-2f76-459e-af36-526b41639c9c","Type":"ContainerDied","Data":"55cbcddcaf71a8a562142db6823a5025be84f22a4c0dcad6c2abb9b13178f9aa"} Jul 03 11:22:33.198363 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.198307 2581 scope.go:117] "RemoveContainer" containerID="765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9" Jul 03 11:22:33.208175 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.208157 2581 scope.go:117] "RemoveContainer" containerID="765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9" Jul 03 11:22:33.208499 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:22:33.208475 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9\": container with ID starting with 765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9 not found: ID does not exist" containerID="765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9" Jul 03 11:22:33.208569 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.208512 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9"} err="failed to get container status \"765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9\": rpc error: code = NotFound desc = could not find container \"765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9\": container with ID starting with 765c2f0e25fd7cd6f98a939463ced9b43af3b283b6683c6d1a896bf2bc4223c9 not found: ID does not exist" Jul 03 11:22:33.219887 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.219852 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7588b74558-xf6hd"] Jul 03 11:22:33.223446 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.223404 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7588b74558-xf6hd"] Jul 03 11:22:33.994521 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:33.994479 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b52e3d1c-2f76-459e-af36-526b41639c9c" path="/var/lib/kubelet/pods/b52e3d1c-2f76-459e-af36-526b41639c9c/volumes" Jul 03 11:22:52.334426 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.334326 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-api-67cbcb4bf6-l8qrn"] Jul 03 11:22:52.334857 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.334774 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52e3d1c-2f76-459e-af36-526b41639c9c" containerName="manager" Jul 03 11:22:52.334857 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.334792 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52e3d1c-2f76-459e-af36-526b41639c9c" containerName="manager" Jul 03 11:22:52.334956 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.334861 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52e3d1c-2f76-459e-af36-526b41639c9c" containerName="manager" Jul 03 11:22:52.337834 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.337814 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.341369 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.341345 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-dockercfg-7w4k6\"" Jul 03 11:22:52.341369 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.341361 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-api-serving-cert\"" Jul 03 11:22:52.359188 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.359155 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/144f2608-8dbd-4b4d-80f6-3fea030d31db-openshift-service-ca\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.359369 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.359214 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sht4m\" (UniqueName: \"kubernetes.io/projected/144f2608-8dbd-4b4d-80f6-3fea030d31db-kube-api-access-sht4m\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.359369 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.359306 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144f2608-8dbd-4b4d-80f6-3fea030d31db-maas-api-tls\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.359988 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.359968 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-67cbcb4bf6-l8qrn"] Jul 03 11:22:52.460353 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.460309 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/144f2608-8dbd-4b4d-80f6-3fea030d31db-openshift-service-ca\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.460353 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.460359 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sht4m\" (UniqueName: \"kubernetes.io/projected/144f2608-8dbd-4b4d-80f6-3fea030d31db-kube-api-access-sht4m\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.460656 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.460392 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144f2608-8dbd-4b4d-80f6-3fea030d31db-maas-api-tls\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.460656 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:22:52.460509 2581 secret.go:189] Couldn't get secret opendatahub/maas-api-serving-cert: secret "maas-api-serving-cert" not found Jul 03 11:22:52.460656 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:22:52.460565 2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/144f2608-8dbd-4b4d-80f6-3fea030d31db-maas-api-tls podName:144f2608-8dbd-4b4d-80f6-3fea030d31db nodeName:}" failed. No retries permitted until 2026-07-03 11:22:52.960547953 +0000 UTC m=+635.523223443 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/144f2608-8dbd-4b4d-80f6-3fea030d31db-maas-api-tls") pod "maas-api-67cbcb4bf6-l8qrn" (UID: "144f2608-8dbd-4b4d-80f6-3fea030d31db") : secret "maas-api-serving-cert" not found Jul 03 11:22:52.460974 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.460953 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/144f2608-8dbd-4b4d-80f6-3fea030d31db-openshift-service-ca\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.469743 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.469720 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sht4m\" (UniqueName: \"kubernetes.io/projected/144f2608-8dbd-4b4d-80f6-3fea030d31db-kube-api-access-sht4m\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.964981 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.964944 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144f2608-8dbd-4b4d-80f6-3fea030d31db-maas-api-tls\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:52.967475 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:52.967454 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/144f2608-8dbd-4b4d-80f6-3fea030d31db-maas-api-tls\") pod \"maas-api-67cbcb4bf6-l8qrn\" (UID: \"144f2608-8dbd-4b4d-80f6-3fea030d31db\") " pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:53.027372 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.027340 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xsmhv"] Jul 03 11:22:53.031062 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.031037 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:53.035843 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.035818 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-szcjt\"" Jul 03 11:22:53.035966 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.035882 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Jul 03 11:22:53.048870 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.048840 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xsmhv"] Jul 03 11:22:53.166653 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.166612 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kd4n\" (UniqueName: \"kubernetes.io/projected/47ea066a-76c7-49bd-9394-40f2ab77d8c5-kube-api-access-9kd4n\") pod \"payload-processing-58f86c7798-xsmhv\" (UID: \"47ea066a-76c7-49bd-9394-40f2ab77d8c5\") " pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:53.166836 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.166699 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ea066a-76c7-49bd-9394-40f2ab77d8c5-plugins-config-volume\") pod \"payload-processing-58f86c7798-xsmhv\" (UID: \"47ea066a-76c7-49bd-9394-40f2ab77d8c5\") " pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:53.247740 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.247653 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:53.267567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.267540 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ea066a-76c7-49bd-9394-40f2ab77d8c5-plugins-config-volume\") pod \"payload-processing-58f86c7798-xsmhv\" (UID: \"47ea066a-76c7-49bd-9394-40f2ab77d8c5\") " pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:53.267731 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.267630 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kd4n\" (UniqueName: \"kubernetes.io/projected/47ea066a-76c7-49bd-9394-40f2ab77d8c5-kube-api-access-9kd4n\") pod \"payload-processing-58f86c7798-xsmhv\" (UID: \"47ea066a-76c7-49bd-9394-40f2ab77d8c5\") " pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:53.268556 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.268531 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47ea066a-76c7-49bd-9394-40f2ab77d8c5-plugins-config-volume\") pod \"payload-processing-58f86c7798-xsmhv\" (UID: \"47ea066a-76c7-49bd-9394-40f2ab77d8c5\") " pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:53.277013 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.276980 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kd4n\" (UniqueName: \"kubernetes.io/projected/47ea066a-76c7-49bd-9394-40f2ab77d8c5-kube-api-access-9kd4n\") pod \"payload-processing-58f86c7798-xsmhv\" (UID: \"47ea066a-76c7-49bd-9394-40f2ab77d8c5\") " pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:53.341892 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.341852 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:53.392211 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.392178 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-api-67cbcb4bf6-l8qrn"] Jul 03 11:22:53.394185 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:22:53.394154 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod144f2608_8dbd_4b4d_80f6_3fea030d31db.slice/crio-267d061d04e2a2d2910b55f353be13c3870ce4ddb0c36a93cf6e699569f110c0 WatchSource:0}: Error finding container 267d061d04e2a2d2910b55f353be13c3870ce4ddb0c36a93cf6e699569f110c0: Status 404 returned error can't find the container with id 267d061d04e2a2d2910b55f353be13c3870ce4ddb0c36a93cf6e699569f110c0 Jul 03 11:22:53.503470 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:53.503410 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-xsmhv"] Jul 03 11:22:53.506132 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:22:53.506102 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ea066a_76c7_49bd_9394_40f2ab77d8c5.slice/crio-9dfe013f35fcbde9580639d10f218b58e4801ccfa447b64b01a9ae5f450d0946 WatchSource:0}: Error finding container 9dfe013f35fcbde9580639d10f218b58e4801ccfa447b64b01a9ae5f450d0946: Status 404 returned error can't find the container with id 9dfe013f35fcbde9580639d10f218b58e4801ccfa447b64b01a9ae5f450d0946 Jul 03 11:22:54.300316 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:54.300277 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" event={"ID":"47ea066a-76c7-49bd-9394-40f2ab77d8c5","Type":"ContainerStarted","Data":"9dfe013f35fcbde9580639d10f218b58e4801ccfa447b64b01a9ae5f450d0946"} Jul 03 11:22:54.301685 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:54.301651 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" event={"ID":"144f2608-8dbd-4b4d-80f6-3fea030d31db","Type":"ContainerStarted","Data":"267d061d04e2a2d2910b55f353be13c3870ce4ddb0c36a93cf6e699569f110c0"} Jul 03 11:22:56.313849 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:56.313812 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" event={"ID":"47ea066a-76c7-49bd-9394-40f2ab77d8c5","Type":"ContainerStarted","Data":"ad6e917d31d9f05dcec02578e2457b1f57b5bf92266343ec2573357b4496cf20"} Jul 03 11:22:56.314333 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:56.313865 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:22:56.315375 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:56.315353 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" event={"ID":"144f2608-8dbd-4b4d-80f6-3fea030d31db","Type":"ContainerStarted","Data":"7d437c81e127152f844eb1314296f6248c1d266a7ca468253b7461659c6c7fe3"} Jul 03 11:22:56.315501 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:56.315472 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:22:56.387713 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:56.387656 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" podStartSLOduration=2.275937821 podStartE2EDuration="4.387637382s" totalImagesPullingTime="2.111699561s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:22:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:22:56.386634351 +0000 UTC m=+638.949309862" watchObservedRunningTime="2026-07-03 11:22:56.387637382 +0000 UTC m=+638.950312895" Jul 03 11:22:56.387891 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:22:56.387743 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" podStartSLOduration=1.391227931 podStartE2EDuration="3.38773671s" totalImagesPullingTime="1.996508779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:22:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:22:56.340385578 +0000 UTC m=+638.903061093" watchObservedRunningTime="2026-07-03 11:22:56.38773671 +0000 UTC m=+638.950412221" Jul 03 11:23:02.325374 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:02.325344 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-api-67cbcb4bf6-l8qrn" Jul 03 11:23:07.322365 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:07.322333 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-xsmhv" Jul 03 11:23:23.500561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.500525 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb"] Jul 03 11:23:23.505101 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.505082 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.508669 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.508646 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-6qpt2\"" Jul 03 11:23:23.508801 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.508690 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Jul 03 11:23:23.508801 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.508703 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Jul 03 11:23:23.508801 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.508690 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" Jul 03 11:23:23.513038 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.512867 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb"] Jul 03 11:23:23.638191 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.638161 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.638191 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.638194 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.638460 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.638216 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028901af-a83d-4d31-b1bb-41cefa0be0d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.638460 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.638235 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7nqh\" (UniqueName: \"kubernetes.io/projected/028901af-a83d-4d31-b1bb-41cefa0be0d5-kube-api-access-d7nqh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.638460 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.638373 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.638460 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.638426 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.738918 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.738879 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.738918 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.738924 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.739134 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.738942 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.739134 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.738957 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028901af-a83d-4d31-b1bb-41cefa0be0d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.739134 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.738974 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7nqh\" (UniqueName: \"kubernetes.io/projected/028901af-a83d-4d31-b1bb-41cefa0be0d5-kube-api-access-d7nqh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.739134 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.739072 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.739351 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.739328 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.739396 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.739362 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.739475 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.739421 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.741536 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.741509 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/028901af-a83d-4d31-b1bb-41cefa0be0d5-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.741705 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.741686 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/028901af-a83d-4d31-b1bb-41cefa0be0d5-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.747821 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.747799 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7nqh\" (UniqueName: \"kubernetes.io/projected/028901af-a83d-4d31-b1bb-41cefa0be0d5-kube-api-access-d7nqh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb\" (UID: \"028901af-a83d-4d31-b1bb-41cefa0be0d5\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.817098 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.816994 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:23.967463 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:23.967417 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb"] Jul 03 11:23:23.969809 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:23:23.969781 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod028901af_a83d_4d31_b1bb_41cefa0be0d5.slice/crio-b692136c84dca6b1b246293181a089471f61c330f31b3df21eeec6ac73a0ea45 WatchSource:0}: Error finding container b692136c84dca6b1b246293181a089471f61c330f31b3df21eeec6ac73a0ea45: Status 404 returned error can't find the container with id b692136c84dca6b1b246293181a089471f61c330f31b3df21eeec6ac73a0ea45 Jul 03 11:23:24.458174 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:24.458120 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" event={"ID":"028901af-a83d-4d31-b1bb-41cefa0be0d5","Type":"ContainerStarted","Data":"b692136c84dca6b1b246293181a089471f61c330f31b3df21eeec6ac73a0ea45"} Jul 03 11:23:25.465010 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:25.464972 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" event={"ID":"028901af-a83d-4d31-b1bb-41cefa0be0d5","Type":"ContainerStarted","Data":"e1efe3d7fb7602fa0cca203a8ba42883ebc667c2ad1d530d4c6491bc305b5428"} Jul 03 11:23:25.485325 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:25.485271 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" podStartSLOduration=1.370632208 podStartE2EDuration="2.485255913s" totalImagesPullingTime="1.114623705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:23:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:23:25.484411379 +0000 UTC m=+668.047086891" watchObservedRunningTime="2026-07-03 11:23:25.485255913 +0000 UTC m=+668.047931462" Jul 03 11:23:33.817334 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:33.817283 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:33.817798 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:33.817389 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:33.830129 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:33.830103 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:34.521252 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:34.521221 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb" Jul 03 11:23:35.493185 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.493148 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz"] Jul 03 11:23:35.498799 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.498780 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.501211 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.501193 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Jul 03 11:23:35.510335 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.510309 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz"] Jul 03 11:23:35.653595 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.653554 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.653595 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.653594 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca395e7b-963b-4ee9-933b-2ba765b13d34-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.653836 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.653621 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8xqq\" (UniqueName: \"kubernetes.io/projected/ca395e7b-963b-4ee9-933b-2ba765b13d34-kube-api-access-z8xqq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.653836 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.653700 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.653836 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.653737 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.653836 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.653772 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.754910 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.754825 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.754910 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.754859 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca395e7b-963b-4ee9-933b-2ba765b13d34-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.754910 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.754877 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8xqq\" (UniqueName: \"kubernetes.io/projected/ca395e7b-963b-4ee9-933b-2ba765b13d34-kube-api-access-z8xqq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.754910 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.754899 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.755208 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.755018 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.755208 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.755091 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.755312 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.755257 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.755312 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.755277 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.755411 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.755372 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.757416 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.757392 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ca395e7b-963b-4ee9-933b-2ba765b13d34-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.757631 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.757615 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ca395e7b-963b-4ee9-933b-2ba765b13d34-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.762916 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.762892 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8xqq\" (UniqueName: \"kubernetes.io/projected/ca395e7b-963b-4ee9-933b-2ba765b13d34-kube-api-access-z8xqq\") pod \"e2e-distinct-simulated-kserve-59975d69f9-lw4zz\" (UID: \"ca395e7b-963b-4ee9-933b-2ba765b13d34\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.809250 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.809213 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:35.963856 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:35.963830 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz"] Jul 03 11:23:35.966103 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:23:35.966076 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca395e7b_963b_4ee9_933b_2ba765b13d34.slice/crio-a5b2ca1ada6dd076d04ce547d3346bd2b8642636fa199f71f44e69f390d0b3d1 WatchSource:0}: Error finding container a5b2ca1ada6dd076d04ce547d3346bd2b8642636fa199f71f44e69f390d0b3d1: Status 404 returned error can't find the container with id a5b2ca1ada6dd076d04ce547d3346bd2b8642636fa199f71f44e69f390d0b3d1 Jul 03 11:23:36.523367 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:36.523268 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" event={"ID":"ca395e7b-963b-4ee9-933b-2ba765b13d34","Type":"ContainerStarted","Data":"6623a9b301fd2e45c4c95099eaa64bce47b470fe10fc976b9e3d35e101ddfcca"} Jul 03 11:23:36.523367 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:36.523335 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" event={"ID":"ca395e7b-963b-4ee9-933b-2ba765b13d34","Type":"ContainerStarted","Data":"a5b2ca1ada6dd076d04ce547d3346bd2b8642636fa199f71f44e69f390d0b3d1"} Jul 03 11:23:36.546825 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:36.546767 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" podStartSLOduration=1.360357016 podStartE2EDuration="1.546750378s" totalImagesPullingTime="186.393362ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:23:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:23:36.545338017 +0000 UTC m=+679.108013522" watchObservedRunningTime="2026-07-03 11:23:36.546750378 +0000 UTC m=+679.109425881" Jul 03 11:23:45.810127 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:45.810079 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:45.810663 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:45.810149 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:45.822662 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:45.822636 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:23:46.582971 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:23:46.582943 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-lw4zz" Jul 03 11:24:26.497412 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.497330 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj"] Jul 03 11:24:26.501966 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.501947 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.504536 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.504512 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Jul 03 11:24:26.512122 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.512090 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj"] Jul 03 11:24:26.530654 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.530617 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.530654 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.530652 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.531102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.530676 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc4fe15-762e-486a-9f13-0fd796d038a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.531102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.530741 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmf4f\" (UniqueName: \"kubernetes.io/projected/7fc4fe15-762e-486a-9f13-0fd796d038a2-kube-api-access-cmf4f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.531102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.530816 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.531102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.530883 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.631868 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.631834 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.631868 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.631868 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.632102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.631892 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc4fe15-762e-486a-9f13-0fd796d038a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.632102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.631914 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmf4f\" (UniqueName: \"kubernetes.io/projected/7fc4fe15-762e-486a-9f13-0fd796d038a2-kube-api-access-cmf4f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.632102 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.632036 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.632240 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.632145 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.632308 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.632284 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.632366 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.632341 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.632600 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.632577 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.634335 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.634316 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7fc4fe15-762e-486a-9f13-0fd796d038a2-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.634532 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.634514 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7fc4fe15-762e-486a-9f13-0fd796d038a2-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.641060 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.641035 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmf4f\" (UniqueName: \"kubernetes.io/projected/7fc4fe15-762e-486a-9f13-0fd796d038a2-kube-api-access-cmf4f\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-qbjqj\" (UID: \"7fc4fe15-762e-486a-9f13-0fd796d038a2\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.812803 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.812710 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:26.952881 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.952854 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj"] Jul 03 11:24:26.955630 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:24:26.955601 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fc4fe15_762e_486a_9f13_0fd796d038a2.slice/crio-67e8986cdcf79d6d8ed5c9b3e3d583e70247ef2f3781e89a1acad8e970212992 WatchSource:0}: Error finding container 67e8986cdcf79d6d8ed5c9b3e3d583e70247ef2f3781e89a1acad8e970212992: Status 404 returned error can't find the container with id 67e8986cdcf79d6d8ed5c9b3e3d583e70247ef2f3781e89a1acad8e970212992 Jul 03 11:24:26.956955 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:26.956929 2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Jul 03 11:24:27.772482 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:27.772416 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" event={"ID":"7fc4fe15-762e-486a-9f13-0fd796d038a2","Type":"ContainerStarted","Data":"a1afca43b7e16fb5bc2e3f129f27e5f47efa6935049221ab2dd744bef5b67d67"} Jul 03 11:24:27.772952 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:27.772494 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" event={"ID":"7fc4fe15-762e-486a-9f13-0fd796d038a2","Type":"ContainerStarted","Data":"67e8986cdcf79d6d8ed5c9b3e3d583e70247ef2f3781e89a1acad8e970212992"} Jul 03 11:24:27.793059 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:27.793004 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" podStartSLOduration=1.60540825 podStartE2EDuration="1.792986705s" totalImagesPullingTime="187.578455ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:24:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:24:27.790691387 +0000 UTC m=+730.353366896" watchObservedRunningTime="2026-07-03 11:24:27.792986705 +0000 UTC m=+730.355662216" Jul 03 11:24:36.813158 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:36.813113 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:36.813685 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:36.813635 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:36.826555 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:36.826527 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:24:37.832977 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:24:37.832946 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-qbjqj" Jul 03 11:27:00.858297 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:00.858260 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b6bc6f48f-76hq7"] Jul 03 11:27:00.861912 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:00.861893 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:00.866767 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:00.866740 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-mvflh\"" Jul 03 11:27:00.866872 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:00.866769 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Jul 03 11:27:00.866872 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:00.866740 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Jul 03 11:27:00.874187 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:00.874167 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b6bc6f48f-76hq7"] Jul 03 11:27:01.013056 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.012999 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f83bcc8-6026-47ef-a52c-40a7c5b3515a-webhook-cert\") pod \"maas-controller-7b6bc6f48f-76hq7\" (UID: \"7f83bcc8-6026-47ef-a52c-40a7c5b3515a\") " pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:01.013056 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.013062 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc9mz\" (UniqueName: \"kubernetes.io/projected/7f83bcc8-6026-47ef-a52c-40a7c5b3515a-kube-api-access-nc9mz\") pod \"maas-controller-7b6bc6f48f-76hq7\" (UID: \"7f83bcc8-6026-47ef-a52c-40a7c5b3515a\") " pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:01.113741 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.113657 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f83bcc8-6026-47ef-a52c-40a7c5b3515a-webhook-cert\") pod \"maas-controller-7b6bc6f48f-76hq7\" (UID: \"7f83bcc8-6026-47ef-a52c-40a7c5b3515a\") " pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:01.113741 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.113696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc9mz\" (UniqueName: \"kubernetes.io/projected/7f83bcc8-6026-47ef-a52c-40a7c5b3515a-kube-api-access-nc9mz\") pod \"maas-controller-7b6bc6f48f-76hq7\" (UID: \"7f83bcc8-6026-47ef-a52c-40a7c5b3515a\") " pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:01.116157 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.116133 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7f83bcc8-6026-47ef-a52c-40a7c5b3515a-webhook-cert\") pod \"maas-controller-7b6bc6f48f-76hq7\" (UID: \"7f83bcc8-6026-47ef-a52c-40a7c5b3515a\") " pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:01.124042 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.124018 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc9mz\" (UniqueName: \"kubernetes.io/projected/7f83bcc8-6026-47ef-a52c-40a7c5b3515a-kube-api-access-nc9mz\") pod \"maas-controller-7b6bc6f48f-76hq7\" (UID: \"7f83bcc8-6026-47ef-a52c-40a7c5b3515a\") " pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:01.173457 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.173388 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:01.312488 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.312459 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b6bc6f48f-76hq7"] Jul 03 11:27:01.314838 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:27:01.314803 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f83bcc8_6026_47ef_a52c_40a7c5b3515a.slice/crio-24075d4154473114c98058f380eec5d3f3aee943082de9989a79433009a46181 WatchSource:0}: Error finding container 24075d4154473114c98058f380eec5d3f3aee943082de9989a79433009a46181: Status 404 returned error can't find the container with id 24075d4154473114c98058f380eec5d3f3aee943082de9989a79433009a46181 Jul 03 11:27:01.539968 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:01.539934 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" event={"ID":"7f83bcc8-6026-47ef-a52c-40a7c5b3515a","Type":"ContainerStarted","Data":"24075d4154473114c98058f380eec5d3f3aee943082de9989a79433009a46181"} Jul 03 11:27:02.546416 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:02.546371 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" event={"ID":"7f83bcc8-6026-47ef-a52c-40a7c5b3515a","Type":"ContainerStarted","Data":"dfd8cd50971cd9914d36a29001464660b788664d63c3b0e30edef12eabb054b8"} Jul 03 11:27:02.546806 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:02.546453 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:02.566285 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:02.566232 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" podStartSLOduration=2.094765045 podStartE2EDuration="2.566204219s" totalImagesPullingTime="471.439174ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:27:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:27:02.564934446 +0000 UTC m=+885.127609960" watchObservedRunningTime="2026-07-03 11:27:02.566204219 +0000 UTC m=+885.128879730" Jul 03 11:27:13.558551 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:13.558462 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b6bc6f48f-76hq7" Jul 03 11:27:17.940167 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:17.940134 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:27:17.940860 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:17.940841 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:27:17.949910 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:17.949873 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:27:17.950143 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:27:17.950121 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:32:17.975305 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:32:17.975268 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:32:17.975996 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:32:17.975977 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:32:17.982773 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:32:17.982754 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:32:17.983220 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:32:17.983199 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:37:18.006706 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:37:18.006679 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:37:18.008987 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:37:18.008965 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:37:18.013588 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:37:18.013571 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:37:18.015938 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:37:18.015921 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:42:18.039311 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:18.039280 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:42:18.043161 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:18.043133 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:42:18.047193 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:18.047166 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:42:18.050824 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:18.050799 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:42:56.210785 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.210733 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849"] Jul 03 11:42:56.214470 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.214425 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.217307 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.217282 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-83f02be8\"/\"e2e-lifecycle-model-83f02be8-kserve-self-signed-certs\"" Jul 03 11:42:56.218402 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.218377 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-83f02be8\"/\"kube-root-ca.crt\"" Jul 03 11:42:56.218561 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.218450 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-83f02be8\"/\"openshift-service-ca.crt\"" Jul 03 11:42:56.218632 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.218576 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-83f02be8\"/\"default-dockercfg-psp6p\"" Jul 03 11:42:56.235313 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.235283 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849"] Jul 03 11:42:56.259231 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.259197 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrrk5\" (UniqueName: \"kubernetes.io/projected/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-kube-api-access-mrrk5\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.259231 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.259229 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-dshm\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.259522 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.259250 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-model-cache\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.259522 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.259298 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-home\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.259522 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.259329 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tls-certs\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.259522 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.259394 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tmp-dir\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360451 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360402 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tmp-dir\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360631 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360520 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrrk5\" (UniqueName: \"kubernetes.io/projected/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-kube-api-access-mrrk5\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360631 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360542 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-dshm\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360631 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360562 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-model-cache\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360631 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360605 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-home\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360795 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360653 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tls-certs\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360872 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360844 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tmp-dir\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360987 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360958 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-model-cache\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.360987 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.360983 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-home\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.363010 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.362991 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-dshm\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.363278 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.363258 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tls-certs\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.370596 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.370571 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrrk5\" (UniqueName: \"kubernetes.io/projected/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-kube-api-access-mrrk5\") pod \"e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.526369 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.526268 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:42:56.680828 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:56.680804 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849"] Jul 03 11:42:56.682745 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:42:56.682716 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0d2b19e_6db0_425b_8399_b72e47c5fb5a.slice/crio-011406cadc4a07895f46dcebd3fd74e84f2206d6e7117f0fed4e6dcf35420008 WatchSource:0}: Error finding container 011406cadc4a07895f46dcebd3fd74e84f2206d6e7117f0fed4e6dcf35420008: Status 404 returned error can't find the container with id 011406cadc4a07895f46dcebd3fd74e84f2206d6e7117f0fed4e6dcf35420008 Jul 03 11:42:57.314422 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:57.314382 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" event={"ID":"a0d2b19e-6db0-425b-8399-b72e47c5fb5a","Type":"ContainerStarted","Data":"e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2"} Jul 03 11:42:57.314897 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:57.314459 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" event={"ID":"a0d2b19e-6db0-425b-8399-b72e47c5fb5a","Type":"ContainerStarted","Data":"011406cadc4a07895f46dcebd3fd74e84f2206d6e7117f0fed4e6dcf35420008"} Jul 03 11:42:57.337527 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:42:57.337459 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" podStartSLOduration=1.337411493 podStartE2EDuration="1.337411493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:42:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:42:57.335854444 +0000 UTC m=+1839.898529969" watchObservedRunningTime="2026-07-03 11:42:57.337411493 +0000 UTC m=+1839.900087005" Jul 03 11:43:06.527350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:43:06.527310 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:43:06.527350 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:43:06.527361 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:43:06.540270 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:43:06.540246 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:43:07.376359 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:43:07.376330 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:43:46.040255 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:43:46.040167 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849"] Jul 03 11:44:01.061067 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.061007 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" podUID="a0d2b19e-6db0-425b-8399-b72e47c5fb5a" containerName="main" containerID="cri-o://e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2" gracePeriod=45 Jul 03 11:44:01.315318 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.315246 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:44:01.492572 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.492532 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-model-cache\") pod \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " Jul 03 11:44:01.492572 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.492578 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-home\") pod \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " Jul 03 11:44:01.492844 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.492615 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mrrk5\" (UniqueName: \"kubernetes.io/projected/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-kube-api-access-mrrk5\") pod \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " Jul 03 11:44:01.492844 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.492685 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-dshm\") pod \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " Jul 03 11:44:01.492844 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.492723 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tls-certs\") pod \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " Jul 03 11:44:01.492844 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.492750 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tmp-dir\") pod \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\" (UID: \"a0d2b19e-6db0-425b-8399-b72e47c5fb5a\") " Jul 03 11:44:01.493046 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.492896 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-model-cache" (OuterVolumeSpecName: "model-cache") pod "a0d2b19e-6db0-425b-8399-b72e47c5fb5a" (UID: "a0d2b19e-6db0-425b-8399-b72e47c5fb5a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:44:01.493046 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.492910 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-home" (OuterVolumeSpecName: "home") pod "a0d2b19e-6db0-425b-8399-b72e47c5fb5a" (UID: "a0d2b19e-6db0-425b-8399-b72e47c5fb5a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:44:01.493152 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.493078 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a0d2b19e-6db0-425b-8399-b72e47c5fb5a" (UID: "a0d2b19e-6db0-425b-8399-b72e47c5fb5a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:44:01.493152 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.493110 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-model-cache\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:44:01.493152 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.493129 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-home\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:44:01.495172 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.495137 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-dshm" (OuterVolumeSpecName: "dshm") pod "a0d2b19e-6db0-425b-8399-b72e47c5fb5a" (UID: "a0d2b19e-6db0-425b-8399-b72e47c5fb5a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:44:01.495289 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.495143 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a0d2b19e-6db0-425b-8399-b72e47c5fb5a" (UID: "a0d2b19e-6db0-425b-8399-b72e47c5fb5a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:44:01.495289 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.495191 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-kube-api-access-mrrk5" (OuterVolumeSpecName: "kube-api-access-mrrk5") pod "a0d2b19e-6db0-425b-8399-b72e47c5fb5a" (UID: "a0d2b19e-6db0-425b-8399-b72e47c5fb5a"). InnerVolumeSpecName "kube-api-access-mrrk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:44:01.593908 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.593816 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrrk5\" (UniqueName: \"kubernetes.io/projected/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-kube-api-access-mrrk5\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:44:01.593908 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.593851 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-dshm\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:44:01.593908 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.593862 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tls-certs\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:44:01.593908 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.593871 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0d2b19e-6db0-425b-8399-b72e47c5fb5a-tmp-dir\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:44:01.638054 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.638015 2581 generic.go:356] "Generic (PLEG): container finished" podID="a0d2b19e-6db0-425b-8399-b72e47c5fb5a" containerID="e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2" exitCode=0 Jul 03 11:44:01.638221 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.638064 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" Jul 03 11:44:01.638221 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.638084 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" event={"ID":"a0d2b19e-6db0-425b-8399-b72e47c5fb5a","Type":"ContainerDied","Data":"e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2"} Jul 03 11:44:01.638221 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.638116 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849" event={"ID":"a0d2b19e-6db0-425b-8399-b72e47c5fb5a","Type":"ContainerDied","Data":"011406cadc4a07895f46dcebd3fd74e84f2206d6e7117f0fed4e6dcf35420008"} Jul 03 11:44:01.638221 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.638134 2581 scope.go:117] "RemoveContainer" containerID="e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2" Jul 03 11:44:01.647946 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.647928 2581 scope.go:117] "RemoveContainer" containerID="e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2" Jul 03 11:44:01.648254 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:44:01.648233 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2\": container with ID starting with e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2 not found: ID does not exist" containerID="e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2" Jul 03 11:44:01.648322 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.648267 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2"} err="failed to get container status \"e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2\": rpc error: code = NotFound desc = could not find container \"e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2\": container with ID starting with e8fcb9873376a0c0e715b6343b6866b0b8b54ac6a8edb4ed7527a0acb9e4b7b2 not found: ID does not exist" Jul 03 11:44:01.659993 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.659960 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849"] Jul 03 11:44:01.666621 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.666591 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-83f02be8/e2e-lifecycle-model-83f02be8-kserve-64fbdb5b87-fl849"] Jul 03 11:44:01.994772 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:44:01.994739 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0d2b19e-6db0-425b-8399-b72e47c5fb5a" path="/var/lib/kubelet/pods/a0d2b19e-6db0-425b-8399-b72e47c5fb5a/volumes" Jul 03 11:47:18.072138 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:47:18.072109 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:47:18.076897 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:47:18.076876 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:47:18.079622 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:47:18.079603 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:47:18.084322 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:47:18.084304 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:49:30.526159 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.526127 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg"] Jul 03 11:49:30.526700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.526562 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0d2b19e-6db0-425b-8399-b72e47c5fb5a" containerName="main" Jul 03 11:49:30.526700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.526573 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0d2b19e-6db0-425b-8399-b72e47c5fb5a" containerName="main" Jul 03 11:49:30.526700 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.526653 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0d2b19e-6db0-425b-8399-b72e47c5fb5a" containerName="main" Jul 03 11:49:30.529847 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.529831 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.534801 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.534779 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f7a699\"/\"kube-root-ca.crt\"" Jul 03 11:49:30.535802 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.535778 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f7a699\"/\"default-dockercfg-m9qk6\"" Jul 03 11:49:30.535946 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.535841 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f7a699\"/\"openshift-service-ca.crt\"" Jul 03 11:49:30.535946 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.535874 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f7a699\"/\"auth-test-model-f7a699-kserve-self-signed-certs\"" Jul 03 11:49:30.546110 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.546081 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg"] Jul 03 11:49:30.617972 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.617932 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-dshm\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.617972 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.617972 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93287a07-0cdd-48ec-aba7-c233279e80b4-tls-certs\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.618230 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.617991 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnm8g\" (UniqueName: \"kubernetes.io/projected/93287a07-0cdd-48ec-aba7-c233279e80b4-kube-api-access-lnm8g\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.618230 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.618102 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-home\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.618230 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.618157 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-tmp-dir\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.618230 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.618223 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-model-cache\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.719637 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.719577 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-model-cache\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.719851 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.719696 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-dshm\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.719851 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.719724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93287a07-0cdd-48ec-aba7-c233279e80b4-tls-certs\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.719851 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.719746 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnm8g\" (UniqueName: \"kubernetes.io/projected/93287a07-0cdd-48ec-aba7-c233279e80b4-kube-api-access-lnm8g\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.719851 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.719798 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-home\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.719851 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.719842 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-tmp-dir\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.720117 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.720021 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-model-cache\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.720117 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.720113 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-home\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.720232 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.720185 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-tmp-dir\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.722098 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.722073 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-dshm\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.722379 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.722361 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93287a07-0cdd-48ec-aba7-c233279e80b4-tls-certs\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.731706 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.731673 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnm8g\" (UniqueName: \"kubernetes.io/projected/93287a07-0cdd-48ec-aba7-c233279e80b4-kube-api-access-lnm8g\") pod \"auth-test-model-f7a699-kserve-545bcdb59f-bxprg\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.839166 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.839067 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:30.974773 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:30.974746 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg"] Jul 03 11:49:30.976534 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:49:30.976504 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93287a07_0cdd_48ec_aba7_c233279e80b4.slice/crio-7345e9479e71b4489eaa02f075f210ae6f59054049f5b93b6eee05fd443fdb43 WatchSource:0}: Error finding container 7345e9479e71b4489eaa02f075f210ae6f59054049f5b93b6eee05fd443fdb43: Status 404 returned error can't find the container with id 7345e9479e71b4489eaa02f075f210ae6f59054049f5b93b6eee05fd443fdb43 Jul 03 11:49:31.303697 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:31.303660 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" event={"ID":"93287a07-0cdd-48ec-aba7-c233279e80b4","Type":"ContainerStarted","Data":"61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f"} Jul 03 11:49:31.303697 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:31.303706 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" event={"ID":"93287a07-0cdd-48ec-aba7-c233279e80b4","Type":"ContainerStarted","Data":"7345e9479e71b4489eaa02f075f210ae6f59054049f5b93b6eee05fd443fdb43"} Jul 03 11:49:31.326880 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:31.326823 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" podStartSLOduration=1.326805868 podStartE2EDuration="1.326805868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:49:31.325374009 +0000 UTC m=+2233.888049519" watchObservedRunningTime="2026-07-03 11:49:31.326805868 +0000 UTC m=+2233.889481379" Jul 03 11:49:40.839692 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:40.839599 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:40.839692 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:40.839665 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:40.852819 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:40.852785 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:49:41.370477 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:49:41.370426 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:51:28.522396 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:28.522304 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg"] Jul 03 11:51:43.543398 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.543340 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" podUID="93287a07-0cdd-48ec-aba7-c233279e80b4" containerName="main" containerID="cri-o://61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f" gracePeriod=45 Jul 03 11:51:43.800919 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.800849 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:51:43.808578 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808557 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-dshm\") pod \"93287a07-0cdd-48ec-aba7-c233279e80b4\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " Jul 03 11:51:43.808671 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808602 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-home\") pod \"93287a07-0cdd-48ec-aba7-c233279e80b4\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " Jul 03 11:51:43.808671 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808650 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93287a07-0cdd-48ec-aba7-c233279e80b4-tls-certs\") pod \"93287a07-0cdd-48ec-aba7-c233279e80b4\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " Jul 03 11:51:43.808762 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808673 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-model-cache\") pod \"93287a07-0cdd-48ec-aba7-c233279e80b4\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " Jul 03 11:51:43.808762 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808716 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-tmp-dir\") pod \"93287a07-0cdd-48ec-aba7-c233279e80b4\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " Jul 03 11:51:43.808762 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808756 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lnm8g\" (UniqueName: \"kubernetes.io/projected/93287a07-0cdd-48ec-aba7-c233279e80b4-kube-api-access-lnm8g\") pod \"93287a07-0cdd-48ec-aba7-c233279e80b4\" (UID: \"93287a07-0cdd-48ec-aba7-c233279e80b4\") " Jul 03 11:51:43.808947 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808891 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-home" (OuterVolumeSpecName: "home") pod "93287a07-0cdd-48ec-aba7-c233279e80b4" (UID: "93287a07-0cdd-48ec-aba7-c233279e80b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:51:43.809030 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808970 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "93287a07-0cdd-48ec-aba7-c233279e80b4" (UID: "93287a07-0cdd-48ec-aba7-c233279e80b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:51:43.809030 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.808978 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-model-cache" (OuterVolumeSpecName: "model-cache") pod "93287a07-0cdd-48ec-aba7-c233279e80b4" (UID: "93287a07-0cdd-48ec-aba7-c233279e80b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:51:43.809160 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.809087 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-tmp-dir\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:51:43.809160 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.809104 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-home\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:51:43.809160 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.809116 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-model-cache\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:51:43.810980 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.810948 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93287a07-0cdd-48ec-aba7-c233279e80b4-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "93287a07-0cdd-48ec-aba7-c233279e80b4" (UID: "93287a07-0cdd-48ec-aba7-c233279e80b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:51:43.810980 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.810952 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-dshm" (OuterVolumeSpecName: "dshm") pod "93287a07-0cdd-48ec-aba7-c233279e80b4" (UID: "93287a07-0cdd-48ec-aba7-c233279e80b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:51:43.811145 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.811017 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93287a07-0cdd-48ec-aba7-c233279e80b4-kube-api-access-lnm8g" (OuterVolumeSpecName: "kube-api-access-lnm8g") pod "93287a07-0cdd-48ec-aba7-c233279e80b4" (UID: "93287a07-0cdd-48ec-aba7-c233279e80b4"). InnerVolumeSpecName "kube-api-access-lnm8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:51:43.909691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.909659 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnm8g\" (UniqueName: \"kubernetes.io/projected/93287a07-0cdd-48ec-aba7-c233279e80b4-kube-api-access-lnm8g\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:51:43.909691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.909686 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93287a07-0cdd-48ec-aba7-c233279e80b4-dshm\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:51:43.909691 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.909698 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93287a07-0cdd-48ec-aba7-c233279e80b4-tls-certs\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:51:43.974391 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.974354 2581 generic.go:356] "Generic (PLEG): container finished" podID="93287a07-0cdd-48ec-aba7-c233279e80b4" containerID="61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f" exitCode=0 Jul 03 11:51:43.974586 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.974405 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" Jul 03 11:51:43.974586 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.974449 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" event={"ID":"93287a07-0cdd-48ec-aba7-c233279e80b4","Type":"ContainerDied","Data":"61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f"} Jul 03 11:51:43.974586 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.974493 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg" event={"ID":"93287a07-0cdd-48ec-aba7-c233279e80b4","Type":"ContainerDied","Data":"7345e9479e71b4489eaa02f075f210ae6f59054049f5b93b6eee05fd443fdb43"} Jul 03 11:51:43.974586 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.974516 2581 scope.go:117] "RemoveContainer" containerID="61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f" Jul 03 11:51:43.984372 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.984347 2581 scope.go:117] "RemoveContainer" containerID="61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f" Jul 03 11:51:43.984723 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:51:43.984702 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f\": container with ID starting with 61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f not found: ID does not exist" containerID="61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f" Jul 03 11:51:43.984829 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:43.984731 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f"} err="failed to get container status \"61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f\": rpc error: code = NotFound desc = could not find container \"61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f\": container with ID starting with 61eeee0cd917075ac671e01459372dddaabe507d540c608b5fe0e79da9819f8f not found: ID does not exist" Jul 03 11:51:44.014114 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:44.014070 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg"] Jul 03 11:51:44.023570 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:44.023540 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/auth-test-model-f7a699-kserve-545bcdb59f-bxprg"] Jul 03 11:51:45.994491 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:51:45.994451 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93287a07-0cdd-48ec-aba7-c233279e80b4" path="/var/lib/kubelet/pods/93287a07-0cdd-48ec-aba7-c233279e80b4/volumes" Jul 03 11:52:18.103609 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:52:18.103576 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:52:18.110908 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:52:18.110879 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:52:18.111670 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:52:18.111652 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:52:18.118535 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:52:18.118516 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-47drl_5a62c8c1-f2e5-4a9a-bda4-530482895a9a/ovn-acl-logging/0.log" Jul 03 11:53:28.309623 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.309584 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm"] Jul 03 11:53:28.310101 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.310006 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93287a07-0cdd-48ec-aba7-c233279e80b4" containerName="main" Jul 03 11:53:28.310101 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.310018 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="93287a07-0cdd-48ec-aba7-c233279e80b4" containerName="main" Jul 03 11:53:28.310101 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.310098 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="93287a07-0cdd-48ec-aba7-c233279e80b4" containerName="main" Jul 03 11:53:28.313194 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.313174 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.317224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.317196 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f7a699\"/\"default-dockercfg-m9qk6\"" Jul 03 11:53:28.317367 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.317277 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-f7a699\"/\"rate-test-model-f7a699-kserve-self-signed-certs\"" Jul 03 11:53:28.317367 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.317336 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f7a699\"/\"kube-root-ca.crt\"" Jul 03 11:53:28.317514 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.317215 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-f7a699\"/\"openshift-service-ca.crt\"" Jul 03 11:53:28.323915 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.323888 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm"] Jul 03 11:53:28.415014 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.414983 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-tmp-dir\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.415205 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.415037 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb223d8-ab39-4012-94dd-01473c241fe1-tls-certs\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.415205 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.415065 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-dshm\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.415205 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.415094 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-home\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.415205 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.415116 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l84db\" (UniqueName: \"kubernetes.io/projected/0eb223d8-ab39-4012-94dd-01473c241fe1-kube-api-access-l84db\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.415205 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.415191 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-model-cache\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.516684 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.516638 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l84db\" (UniqueName: \"kubernetes.io/projected/0eb223d8-ab39-4012-94dd-01473c241fe1-kube-api-access-l84db\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.516684 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.516686 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-model-cache\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.516939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.516724 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-tmp-dir\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.516939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.516776 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb223d8-ab39-4012-94dd-01473c241fe1-tls-certs\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.516939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.516795 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-dshm\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.516939 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.516824 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-home\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.517210 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.517183 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-model-cache\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.517280 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.517220 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-tmp-dir\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.517280 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.517190 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-home\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.519336 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.519307 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-dshm\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.519528 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.519508 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb223d8-ab39-4012-94dd-01473c241fe1-tls-certs\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.527277 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.527249 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l84db\" (UniqueName: \"kubernetes.io/projected/0eb223d8-ab39-4012-94dd-01473c241fe1-kube-api-access-l84db\") pod \"rate-test-model-f7a699-kserve-7994bcc84c-7k7vm\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.625209 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.625117 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:28.776330 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:28.776291 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm"] Jul 03 11:53:28.777885 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:53:28.777851 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0eb223d8_ab39_4012_94dd_01473c241fe1.slice/crio-b3e43f79f5810630bec3311ddb1854e8038dce77c576f8758ee444ecff2a9265 WatchSource:0}: Error finding container b3e43f79f5810630bec3311ddb1854e8038dce77c576f8758ee444ecff2a9265: Status 404 returned error can't find the container with id b3e43f79f5810630bec3311ddb1854e8038dce77c576f8758ee444ecff2a9265 Jul 03 11:53:29.497559 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:29.497526 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" event={"ID":"0eb223d8-ab39-4012-94dd-01473c241fe1","Type":"ContainerStarted","Data":"6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80"} Jul 03 11:53:29.497973 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:29.497567 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" event={"ID":"0eb223d8-ab39-4012-94dd-01473c241fe1","Type":"ContainerStarted","Data":"b3e43f79f5810630bec3311ddb1854e8038dce77c576f8758ee444ecff2a9265"} Jul 03 11:53:29.521721 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:29.521663 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" podStartSLOduration=1.52164427 podStartE2EDuration="1.52164427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-03 11:53:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:53:29.520637011 +0000 UTC m=+2472.083312543" watchObservedRunningTime="2026-07-03 11:53:29.52164427 +0000 UTC m=+2472.084319782" Jul 03 11:53:38.625846 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:38.625795 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:38.626348 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:38.625872 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:38.638567 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:38.638538 2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:53:39.558715 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:53:39.558685 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:54:18.822549 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:18.822457 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm"] Jul 03 11:54:33.842930 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:33.842887 2581 kuberuntime_container.go:872] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" podUID="0eb223d8-ab39-4012-94dd-01473c241fe1" containerName="main" containerID="cri-o://6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80" gracePeriod=45 Jul 03 11:54:34.109652 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.109621 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:54:34.154895 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.154858 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-home\") pod \"0eb223d8-ab39-4012-94dd-01473c241fe1\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " Jul 03 11:54:34.154895 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.154900 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-tmp-dir\") pod \"0eb223d8-ab39-4012-94dd-01473c241fe1\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " Jul 03 11:54:34.155162 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.154941 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-dshm\") pod \"0eb223d8-ab39-4012-94dd-01473c241fe1\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " Jul 03 11:54:34.155162 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.154973 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-model-cache\") pod \"0eb223d8-ab39-4012-94dd-01473c241fe1\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " Jul 03 11:54:34.155162 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.155064 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l84db\" (UniqueName: \"kubernetes.io/projected/0eb223d8-ab39-4012-94dd-01473c241fe1-kube-api-access-l84db\") pod \"0eb223d8-ab39-4012-94dd-01473c241fe1\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " Jul 03 11:54:34.155162 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.155122 2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb223d8-ab39-4012-94dd-01473c241fe1-tls-certs\") pod \"0eb223d8-ab39-4012-94dd-01473c241fe1\" (UID: \"0eb223d8-ab39-4012-94dd-01473c241fe1\") " Jul 03 11:54:34.155368 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.155202 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-home" (OuterVolumeSpecName: "home") pod "0eb223d8-ab39-4012-94dd-01473c241fe1" (UID: "0eb223d8-ab39-4012-94dd-01473c241fe1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:54:34.155368 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.155301 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0eb223d8-ab39-4012-94dd-01473c241fe1" (UID: "0eb223d8-ab39-4012-94dd-01473c241fe1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:54:34.155368 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.155342 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0eb223d8-ab39-4012-94dd-01473c241fe1" (UID: "0eb223d8-ab39-4012-94dd-01473c241fe1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:54:34.155629 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.155612 2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-home\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:54:34.155742 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.155656 2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-tmp-dir\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:54:34.155742 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.155671 2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-model-cache\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:54:34.157354 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.157318 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-dshm" (OuterVolumeSpecName: "dshm") pod "0eb223d8-ab39-4012-94dd-01473c241fe1" (UID: "0eb223d8-ab39-4012-94dd-01473c241fe1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Jul 03 11:54:34.157852 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.157821 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0eb223d8-ab39-4012-94dd-01473c241fe1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0eb223d8-ab39-4012-94dd-01473c241fe1" (UID: "0eb223d8-ab39-4012-94dd-01473c241fe1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Jul 03 11:54:34.157852 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.157840 2581 operation_generator.go:781] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eb223d8-ab39-4012-94dd-01473c241fe1-kube-api-access-l84db" (OuterVolumeSpecName: "kube-api-access-l84db") pod "0eb223d8-ab39-4012-94dd-01473c241fe1" (UID: "0eb223d8-ab39-4012-94dd-01473c241fe1"). InnerVolumeSpecName "kube-api-access-l84db". PluginName "kubernetes.io/projected", VolumeGIDValue "" Jul 03 11:54:34.257171 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.257128 2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0eb223d8-ab39-4012-94dd-01473c241fe1-dshm\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:54:34.257171 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.257165 2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l84db\" (UniqueName: \"kubernetes.io/projected/0eb223d8-ab39-4012-94dd-01473c241fe1-kube-api-access-l84db\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:54:34.257171 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.257176 2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0eb223d8-ab39-4012-94dd-01473c241fe1-tls-certs\") on node \"ip-10-0-141-89.ec2.internal\" DevicePath \"\"" Jul 03 11:54:34.829599 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.829561 2581 generic.go:356] "Generic (PLEG): container finished" podID="0eb223d8-ab39-4012-94dd-01473c241fe1" containerID="6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80" exitCode=0 Jul 03 11:54:34.829783 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.829610 2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" Jul 03 11:54:34.829783 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.829654 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" event={"ID":"0eb223d8-ab39-4012-94dd-01473c241fe1","Type":"ContainerDied","Data":"6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80"} Jul 03 11:54:34.829783 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.829699 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm" event={"ID":"0eb223d8-ab39-4012-94dd-01473c241fe1","Type":"ContainerDied","Data":"b3e43f79f5810630bec3311ddb1854e8038dce77c576f8758ee444ecff2a9265"} Jul 03 11:54:34.829783 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.829725 2581 scope.go:117] "RemoveContainer" containerID="6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80" Jul 03 11:54:34.846511 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.846316 2581 scope.go:117] "RemoveContainer" containerID="6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80" Jul 03 11:54:34.846758 ip-10-0-141-89 kubenswrapper[2581]: E0703 11:54:34.846631 2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80\": container with ID starting with 6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80 not found: ID does not exist" containerID="6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80" Jul 03 11:54:34.846758 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.846666 2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80"} err="failed to get container status \"6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80\": rpc error: code = NotFound desc = could not find container \"6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80\": container with ID starting with 6325ba290813e3d90297202edb291bc04b86513fa058ec15e6f66734418c6f80 not found: ID does not exist" Jul 03 11:54:34.857900 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.857864 2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm"] Jul 03 11:54:34.859646 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:34.859622 2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-f7a699/rate-test-model-f7a699-kserve-7994bcc84c-7k7vm"] Jul 03 11:54:35.994696 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:54:35.994662 2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0eb223d8-ab39-4012-94dd-01473c241fe1" path="/var/lib/kubelet/pods/0eb223d8-ab39-4012-94dd-01473c241fe1/volumes" Jul 03 11:55:15.963179 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:15.963149 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-67cbcb4bf6-l8qrn_144f2608-8dbd-4b4d-80f6-3fea030d31db/maas-api/0.log" Jul 03 11:55:16.298224 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:16.298136 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b6bc6f48f-76hq7_7f83bcc8-6026-47ef-a52c-40a7c5b3515a/manager/0.log" Jul 03 11:55:16.788688 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:16.788660 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b8c56cd69-ftb5d_65a523fa-70f8-48bf-8ad3-5ca0575733e8/copy-manifests/0.log" Jul 03 11:55:16.796151 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:16.796129 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b8c56cd69-ftb5d_65a523fa-70f8-48bf-8ad3-5ca0575733e8/manager/0.log" Jul 03 11:55:17.032948 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:17.032919 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-9hpmt_873236fb-b94a-4729-acf6-ada0a1fa5133/postgres/0.log" Jul 03 11:55:18.708334 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:18.708303 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lfsdk_0b3b5b04-b58e-480f-9ed1-b976724346b2/kuadrant-console-plugin/0.log" Jul 03 11:55:19.897721 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:19.897689 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb49b9d79-vhj2d_04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5/kube-auth-proxy/0.log" Jul 03 11:55:20.269098 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:20.269004 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xsmhv_47ea066a-76c7-49bd-9394-40f2ab77d8c5/payload-processing/0.log" Jul 03 11:55:20.379268 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:20.379236 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678cfbb59c-mm6jx_6ef1ae29-90df-42ac-8b5d-049e00ffe01d/router/0.log" Jul 03 11:55:20.718422 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:20.718395 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-kzlvb_028901af-a83d-4d31-b1bb-41cefa0be0d5/main/0.log" Jul 03 11:55:20.822815 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:20.822783 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-lw4zz_ca395e7b-963b-4ee9-933b-2ba765b13d34/main/0.log" Jul 03 11:55:21.153851 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:21.153823 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-qbjqj_7fc4fe15-762e-486a-9f13-0fd796d038a2/main/0.log" Jul 03 11:55:28.031084 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:28.031027 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-q2wrq_0b814191-d67d-47b0-8c3f-af739872704b/global-pull-secret-syncer/0.log" Jul 03 11:55:28.210880 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:28.210847 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wkzwc_051916bd-975d-49bf-810c-768ed087aa68/konnectivity-agent/0.log" Jul 03 11:55:28.297555 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:28.297474 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-89.ec2.internal_36c40a12ef0a62b4aa40697a8c2cbf13/haproxy/0.log" Jul 03 11:55:33.916978 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:33.916949 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-lfsdk_0b3b5b04-b58e-480f-9ed1-b976724346b2/kuadrant-console-plugin/0.log" Jul 03 11:55:35.847172 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:35.847097 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecf47df-010c-4dea-8449-b96df2e4cbcd/alertmanager/0.log" Jul 03 11:55:35.874087 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:35.874054 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecf47df-010c-4dea-8449-b96df2e4cbcd/config-reloader/0.log" Jul 03 11:55:35.901448 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:35.901404 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecf47df-010c-4dea-8449-b96df2e4cbcd/kube-rbac-proxy-web/0.log" Jul 03 11:55:35.930377 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:35.930345 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecf47df-010c-4dea-8449-b96df2e4cbcd/kube-rbac-proxy/0.log" Jul 03 11:55:35.957132 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:35.957101 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecf47df-010c-4dea-8449-b96df2e4cbcd/kube-rbac-proxy-metric/0.log" Jul 03 11:55:35.985766 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:35.985738 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecf47df-010c-4dea-8449-b96df2e4cbcd/prom-label-proxy/0.log" Jul 03 11:55:36.024628 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.024602 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_3ecf47df-010c-4dea-8449-b96df2e4cbcd/init-config-reloader/0.log" Jul 03 11:55:36.075727 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.075695 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-64786d9cbb-28qt8_950cea78-bd24-478c-810c-e398a8075c7d/cluster-monitoring-operator/0.log" Jul 03 11:55:36.105803 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.105726 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-fd7tm_c6bcf17a-1e79-4923-b249-6370a485725d/kube-state-metrics/0.log" Jul 03 11:55:36.132020 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.131992 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-fd7tm_c6bcf17a-1e79-4923-b249-6370a485725d/kube-rbac-proxy-main/0.log" Jul 03 11:55:36.160515 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.160487 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-7c9dbb8b7d-fd7tm_c6bcf17a-1e79-4923-b249-6370a485725d/kube-rbac-proxy-self/0.log" Jul 03 11:55:36.393542 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.393514 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkdgv_61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc/node-exporter/0.log" Jul 03 11:55:36.421598 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.421568 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkdgv_61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc/kube-rbac-proxy/0.log" Jul 03 11:55:36.456888 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.456863 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kkdgv_61bfe97f-6da4-4ace-b0bb-79f5a72ac2fc/init-textfile/0.log" Jul 03 11:55:36.596616 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.596590 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pld97_d32ab786-7fe5-4249-97b1-b4476fc7d7c0/kube-rbac-proxy-main/0.log" Jul 03 11:55:36.624556 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.624533 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pld97_d32ab786-7fe5-4249-97b1-b4476fc7d7c0/kube-rbac-proxy-self/0.log" Jul 03 11:55:36.650257 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.650172 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-54cf96dfcf-pld97_d32ab786-7fe5-4249-97b1-b4476fc7d7c0/openshift-state-metrics/0.log" Jul 03 11:55:36.693248 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.693217 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f01678ac-2fa9-4e1c-9482-42ae450c6972/prometheus/0.log" Jul 03 11:55:36.719579 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.719553 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f01678ac-2fa9-4e1c-9482-42ae450c6972/config-reloader/0.log" Jul 03 11:55:36.754270 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.754243 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f01678ac-2fa9-4e1c-9482-42ae450c6972/thanos-sidecar/0.log" Jul 03 11:55:36.778745 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.778719 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f01678ac-2fa9-4e1c-9482-42ae450c6972/kube-rbac-proxy-web/0.log" Jul 03 11:55:36.806930 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.806901 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f01678ac-2fa9-4e1c-9482-42ae450c6972/kube-rbac-proxy/0.log" Jul 03 11:55:36.832211 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.832186 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f01678ac-2fa9-4e1c-9482-42ae450c6972/kube-rbac-proxy-thanos/0.log" Jul 03 11:55:36.860112 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.860088 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_f01678ac-2fa9-4e1c-9482-42ae450c6972/init-config-reloader/0.log" Jul 03 11:55:36.944266 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:36.944187 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-d559d66d5-9px46_a4a585b5-9cc0-494d-bf0e-025e2e95c227/prometheus-operator-admission-webhook/0.log" Jul 03 11:55:37.602965 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.602932 2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5"] Jul 03 11:55:37.603372 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.603360 2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0eb223d8-ab39-4012-94dd-01473c241fe1" containerName="main" Jul 03 11:55:37.603420 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.603375 2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0eb223d8-ab39-4012-94dd-01473c241fe1" containerName="main" Jul 03 11:55:37.603506 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.603497 2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0eb223d8-ab39-4012-94dd-01473c241fe1" containerName="main" Jul 03 11:55:37.606655 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.606639 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.609378 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.609358 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fgj8l\"/\"kube-root-ca.crt\"" Jul 03 11:55:37.609507 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.609379 2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fgj8l\"/\"default-dockercfg-z5qdj\"" Jul 03 11:55:37.610408 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.610385 2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fgj8l\"/\"openshift-service-ca.crt\"" Jul 03 11:55:37.614921 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.614893 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5"] Jul 03 11:55:37.748319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.748239 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-proc\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.748319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.748272 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-lib-modules\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.748319 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.748307 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-podres\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.748597 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.748366 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-sys\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.748597 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.748398 2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tjzl\" (UniqueName: \"kubernetes.io/projected/c8060a6c-066b-4854-a473-40f97cca8521-kube-api-access-4tjzl\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849028 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.848996 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tjzl\" (UniqueName: \"kubernetes.io/projected/c8060a6c-066b-4854-a473-40f97cca8521-kube-api-access-4tjzl\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849208 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.849057 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-proc\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849208 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.849121 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-proc\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849208 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.849179 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-lib-modules\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849343 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.849234 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-podres\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849343 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.849286 2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-sys\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849343 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.849298 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-lib-modules\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849510 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.849397 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-sys\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.849510 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.849412 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c8060a6c-066b-4854-a473-40f97cca8521-podres\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.862877 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.862841 2581 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tjzl\" (UniqueName: \"kubernetes.io/projected/c8060a6c-066b-4854-a473-40f97cca8521-kube-api-access-4tjzl\") pod \"perf-node-gather-daemonset-7k5m5\" (UID: \"c8060a6c-066b-4854-a473-40f97cca8521\") " pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:37.918054 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:37.918014 2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:38.064963 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:38.064927 2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5"] Jul 03 11:55:38.068842 ip-10-0-141-89 kubenswrapper[2581]: W0703 11:55:38.068808 2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc8060a6c_066b_4854_a473_40f97cca8521.slice/crio-1ccf3ec51daa21b339a8398b2ea39161b0124495248cd10e1d2fe90ec1277418 WatchSource:0}: Error finding container 1ccf3ec51daa21b339a8398b2ea39161b0124495248cd10e1d2fe90ec1277418: Status 404 returned error can't find the container with id 1ccf3ec51daa21b339a8398b2ea39161b0124495248cd10e1d2fe90ec1277418 Jul 03 11:55:38.157752 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:38.157716 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" event={"ID":"c8060a6c-066b-4854-a473-40f97cca8521","Type":"ContainerStarted","Data":"1ccf3ec51daa21b339a8398b2ea39161b0124495248cd10e1d2fe90ec1277418"} Jul 03 11:55:38.390047 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:38.390015 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6bffdf9778-rcsx9_c9dd690e-80aa-4886-963a-5616b8041655/networking-console-plugin/0.log" Jul 03 11:55:38.952948 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:38.952917 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/1.log" Jul 03 11:55:38.956897 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:38.956878 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-68b757fb67-p2bsb_1c73b128-ad8e-46c4-86bf-1596700247a4/console-operator/2.log" Jul 03 11:55:39.164817 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:39.164783 2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" event={"ID":"c8060a6c-066b-4854-a473-40f97cca8521","Type":"ContainerStarted","Data":"b6c460c1716a10cec0eea7e152527e442c6cbe0476489f9f68f7ee84e4a603b7"} Jul 03 11:55:39.164997 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:39.164925 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:39.182757 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:39.182702 2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" podStartSLOduration=2.182685623 podStartE2EDuration="2.182685623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-03 11:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-03 11:55:39.181277272 +0000 UTC m=+2601.743952784" watchObservedRunningTime="2026-07-03 11:55:39.182685623 +0000 UTC m=+2601.745361134" Jul 03 11:55:40.118316 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:40.118287 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-5d456fdbcd-2j6rg_e7554f43-afeb-4d23-9530-589db05eeb2f/volume-data-source-validator/0.log" Jul 03 11:55:41.325378 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:41.325344 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bfkkq_ae916522-5d55-4fbf-b304-a7f05fae5691/dns/0.log" Jul 03 11:55:41.361786 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:41.361760 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bfkkq_ae916522-5d55-4fbf-b304-a7f05fae5691/kube-rbac-proxy/0.log" Jul 03 11:55:41.458442 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:41.458412 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8d5tj_1a6b54da-ff3a-4ad1-868b-1f06684af304/dns-node-resolver/0.log" Jul 03 11:55:42.141714 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:42.141687 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mc48b_b4cfd292-27df-444a-b0af-43067d41ee34/node-ca/0.log" Jul 03 11:55:43.398366 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:43.398339 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb49b9d79-vhj2d_04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5/kube-auth-proxy/0.log" Jul 03 11:55:43.559724 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:43.559684 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xsmhv_47ea066a-76c7-49bd-9394-40f2ab77d8c5/payload-processing/0.log" Jul 03 11:55:43.585652 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:43.585621 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678cfbb59c-mm6jx_6ef1ae29-90df-42ac-8b5d-049e00ffe01d/router/0.log" Jul 03 11:55:44.318295 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:44.318264 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-rzn28_c5c70232-5339-4786-bf98-86283e335ea1/serve-healthcheck-canary/0.log" Jul 03 11:55:44.836629 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:44.836595 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-7wpk5_4b09b25f-d483-4e17-8c66-76fc0dbc1bf7/insights-operator/0.log" Jul 03 11:55:44.837382 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:44.837364 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-5c847dcfc4-7wpk5_4b09b25f-d483-4e17-8c66-76fc0dbc1bf7/insights-operator/1.log" Jul 03 11:55:44.863425 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:44.863397 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhkvs_6d89b76a-0f00-4802-999b-614c7a0ac325/kube-rbac-proxy/0.log" Jul 03 11:55:44.889032 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:44.889010 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhkvs_6d89b76a-0f00-4802-999b-614c7a0ac325/exporter/0.log" Jul 03 11:55:44.918077 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:44.918054 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qhkvs_6d89b76a-0f00-4802-999b-614c7a0ac325/extractor/0.log" Jul 03 11:55:45.180084 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:45.180056 2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fgj8l/perf-node-gather-daemonset-7k5m5" Jul 03 11:55:47.727751 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:47.727723 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-api-67cbcb4bf6-l8qrn_144f2608-8dbd-4b4d-80f6-3fea030d31db/maas-api/0.log" Jul 03 11:55:47.897354 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:47.897325 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b6bc6f48f-76hq7_7f83bcc8-6026-47ef-a52c-40a7c5b3515a/manager/0.log" Jul 03 11:55:48.144299 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:48.144270 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b8c56cd69-ftb5d_65a523fa-70f8-48bf-8ad3-5ca0575733e8/manager/0.log" Jul 03 11:55:48.169245 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:48.169214 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-b8c56cd69-ftb5d_65a523fa-70f8-48bf-8ad3-5ca0575733e8/copy-manifests/0.log" Jul 03 11:55:48.258906 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:48.258876 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_postgres-69fb4594cf-9hpmt_873236fb-b94a-4729-acf6-ada0a1fa5133/postgres/0.log" Jul 03 11:55:48.546841 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:48.546765 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb49b9d79-vhj2d_04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5/kube-auth-proxy/0.log" Jul 03 11:55:48.661593 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:48.661565 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xsmhv_47ea066a-76c7-49bd-9394-40f2ab77d8c5/payload-processing/0.log" Jul 03 11:55:48.675941 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:48.675915 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678cfbb59c-mm6jx_6ef1ae29-90df-42ac-8b5d-049e00ffe01d/router/0.log" Jul 03 11:55:49.706110 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:49.706081 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-6f9f46b759-sblp2_d2d9a0d9-f891-46e4-9578-b4a2e1e4a85f/manager/0.log" Jul 03 11:55:50.077846 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:50.077817 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5cb49b9d79-vhj2d_04c0c892-9e63-4a3d-8c9a-fc8e87c5f8f5/kube-auth-proxy/0.log" Jul 03 11:55:50.213696 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:50.213659 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-xsmhv_47ea066a-76c7-49bd-9394-40f2ab77d8c5/payload-processing/0.log" Jul 03 11:55:50.229009 ip-10-0-141-89 kubenswrapper[2581]: I0703 11:55:50.228982 2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-678cfbb59c-mm6jx_6ef1ae29-90df-42ac-8b5d-049e00ffe01d/router/0.log"